My brother-in-law, my wife's younger brother, is blind. He's a good programmer though, great brains in there. He uses MS-Windows with JAWS (screen reader software). What he can do with windows is amazing, considering that it's so graphical.
He used to use a 400Mhz PIII with 192MB of RAM. I've inherited that PC since I found a used PC that is much faster (an AMD 1.7Ghz box) and gave that for him to use.
I run linux on the slow box, with icewm for the window manager (I use icewm everywhere, even on the fastest computers because I can't stand the slowness of Gnome and KDE). It runs everything reasonably well except Firefox. It's great to use Firefox on fast boxes, but it's dog slow on a slow box. I've got to look at Opera for that box.
I don't remember running openoffice on that box. Maybe I've done it, I don't use very complex files though, so I wouldn't have been stressing openoffice anyway, even if I ran it. I'm sure it's slow, but it's probably usable. I tend to use gnumeric for spreadsheets and I just don't use a wordprocessor that much though. So on a slow, memory-constrained computer, careful choice of office software can make the computer usable.
The PIII box used to run windows 2000. It was reasonably usable, but I'm not sure how well it compared with linux. It was probably about the same speed, or maybe a little slower (because of the antivirus reading everything before handing data off to applications reading the disk). On the other hand, it was *far* less capable than my current setup (apache, php, C, C++, Java, ruby, scheme, guile, postgres all running on it, plus all the client tools I use).
I've got to find an alternative to firefox though. Or wait until someone comes up with a comprehensive howto on how to minimize the memory and CPU that firefox uses.
Friday, May 26, 2006
Friday, May 05, 2006
philvbdev
I subscribed to the philvbdev yahoogroup to see if I could hire VB developers off there. The company I work for is hiring. After I posted though, I started getting all these emails from Indian freshers. There must be a network (or a bot) where newbie South Indian software developers get email addresses and the email addresses spread like wildfire. It's bad enough that if anyone sends me anything with "fresher" in the subject, I'm not going to see that email at all, it'll be automatically deleted [not even spending any time in the purgatory that is my spam mbox or trash].
It's a waste of time for them to be emailing me because we need the developers for local employment and we're not going to be paying anyone to come over from a foreign country to work here (with the attendant problems with visas and such).
I joined the group because I thought it was a forum for philippine VB developers. It looks like it's been abandoned by filipinos though. Very few posts are about VB problems or solutions. Mostly what I see (what I saw, I've since unsubscribed) are spam posts about certifications, certification questions and training, etc.
It's a waste of time for them to be emailing me because we need the developers for local employment and we're not going to be paying anyone to come over from a foreign country to work here (with the attendant problems with visas and such).
I joined the group because I thought it was a forum for philippine VB developers. It looks like it's been abandoned by filipinos though. Very few posts are about VB problems or solutions. Mostly what I see (what I saw, I've since unsubscribed) are spam posts about certifications, certification questions and training, etc.
Wednesday, March 29, 2006
laptop computer speed
My cousin-in-law asked me last week what I'd choose between a computer (laptop, actually), with a 1.xGhz CPU and 512MB of RAM and a 2.xGhz CPU with 256MB of RAM. I said I'd choose the first.
I've been monitoring CPU use on the laptop I currently use (icewm makes that really easy, with CPU and network monitors on the taskbar), and I just don't see a lot of CPU peaking. There are occasional peaks when a scheduled rsync job runs (copying data over a vpn from the office to my laptop, cheap remote backup), and when i run xboard+gnuchess. When I test one or another program I'm developing I may see some peaking, and sometimes when I run vmware (not a lot of that though).
Mainly, though, my CPU usage stays pretty low.
I recently finally built a kernel that works well with my mobile AMD CPU and now I'm able to set the CPU maximum speed to a large range of speeds (.66Ghz to 1.7Ghz, plus 6 other speeds in between). I've also got the cpufreq/scaling_governor set to ondemand. The computer pretty much ALWAYS stays at the slowest speed all the time except when running xboard+gnuchess or very occasionally, when something CPU intensive runs. When an application needs more CPU, the CPU speed ramps up immediately, and when it's no longer needed, the CPU slows down to .66Ghz.
From the fact that the scaling_max_freq pretty much stays at the slowest speed except for very short periods, my opinion on which laptop to buy is reinforced. The icewm monitor also displays related information (e.g., how much IO is occurring) and I find that the laptop slows down more due to waiting for IO (well, laptop drives are slow) than due to needing more CPU. So more RAM will help with the IO requirements (the OS can cache more of the disk), speeding up the general experience, more than higher CPU speed.
Actually, his original question had to do with a celeron (or some other internal-cache-challenged CPU) versus a full CPU at the same or a higher frequency. I think that RAM is still more important, generally, than CPU speed.
Of course, if the main use of the computer is going to be for photoshop or corel draw effects or CAD and 3D rendering, then I'd go with all the CPU power I could get PLUS all the RAM I could get. but for general use, even for a software developer like me, any 1.3Ghz box with more memory is probably going to be the better deal than a higher frequency box with half the memory.
I've been monitoring CPU use on the laptop I currently use (icewm makes that really easy, with CPU and network monitors on the taskbar), and I just don't see a lot of CPU peaking. There are occasional peaks when a scheduled rsync job runs (copying data over a vpn from the office to my laptop, cheap remote backup), and when i run xboard+gnuchess. When I test one or another program I'm developing I may see some peaking, and sometimes when I run vmware (not a lot of that though).
Mainly, though, my CPU usage stays pretty low.
I recently finally built a kernel that works well with my mobile AMD CPU and now I'm able to set the CPU maximum speed to a large range of speeds (.66Ghz to 1.7Ghz, plus 6 other speeds in between). I've also got the cpufreq/scaling_governor set to ondemand. The computer pretty much ALWAYS stays at the slowest speed all the time except when running xboard+gnuchess or very occasionally, when something CPU intensive runs. When an application needs more CPU, the CPU speed ramps up immediately, and when it's no longer needed, the CPU slows down to .66Ghz.
From the fact that the scaling_max_freq pretty much stays at the slowest speed except for very short periods, my opinion on which laptop to buy is reinforced. The icewm monitor also displays related information (e.g., how much IO is occurring) and I find that the laptop slows down more due to waiting for IO (well, laptop drives are slow) than due to needing more CPU. So more RAM will help with the IO requirements (the OS can cache more of the disk), speeding up the general experience, more than higher CPU speed.
Actually, his original question had to do with a celeron (or some other internal-cache-challenged CPU) versus a full CPU at the same or a higher frequency. I think that RAM is still more important, generally, than CPU speed.
Of course, if the main use of the computer is going to be for photoshop or corel draw effects or CAD and 3D rendering, then I'd go with all the CPU power I could get PLUS all the RAM I could get. but for general use, even for a software developer like me, any 1.3Ghz box with more memory is probably going to be the better deal than a higher frequency box with half the memory.
Wednesday, March 08, 2006
mozilla extensions
I've been wary/skeptical for a long time about mozilla/firefox extensions. Firefox is a memory hog as it is, and there are some hard to isolate weird CPU hogging things in there too. So, while I love the browser, I go out of my way to NOT make it any more memory/cpu hungry. And in general, for complex things that have weird behaviors, I prefer to keep things as simple as possible so that problem isolation is easier, or at least that the chances of tickling a bug are lower.
Late though, and strangely enough, because I saw a link to an article that discussed how to make firefox eat less memory and use less CPU, I followed another link and decided to install a few very conservatively chosen extensions. I'll add links at some point, for now, just discussion.
NoScript is great. I have it block all javascript by default. And then, for sites that need scripts, I turn javascript on either temporarily (current session) or permanently. Very nice since most of the time I don't want javascript and it just slows things down anyway.
AdBlock works pretty well. Some ads get through and it's not convenient to point them out to AdBlock, but all in all, it's great. I wish I could *unblock* google ads. They're not intrusive, no moving graphics and crap like that, and they're generally relevant, although I rarely click. I'll figure out how to do that at some point.
Tab Mix Plus is great. There are lots of features there, most of which I don't know how to use. The main thing for me is that it has tab memory like Opera has. That is, it's possible to configure it so that when the browser starts, it will auto-load the tabs that were there when the browser was last closed. It also saves tab status while browsing, so that firefox and crashes (rare, but not completely eradicated) don't completely wipe me out. If I've downloaded 15 tabs to view, then tab memory will bring them all back up when I restart firefox. It seems to work better than Opera's memory, but I haven't used Opera in 2 years. I'm sure Opera has made their tab memory much more robust by now, although it was already a great and wonderful feature when I used it back then.
PDF download is convenient since I've never been able to stand acrobat reader displaying itself in the browser and then taking up so long to start. Not that I have acrobat reader on this computer anymore, or any computer I use, but the extra control over PDF downloads is nice. This is a relatively low profit item for me though, I may drop it.
Late though, and strangely enough, because I saw a link to an article that discussed how to make firefox eat less memory and use less CPU, I followed another link and decided to install a few very conservatively chosen extensions. I'll add links at some point, for now, just discussion.
NoScript is great. I have it block all javascript by default. And then, for sites that need scripts, I turn javascript on either temporarily (current session) or permanently. Very nice since most of the time I don't want javascript and it just slows things down anyway.
AdBlock works pretty well. Some ads get through and it's not convenient to point them out to AdBlock, but all in all, it's great. I wish I could *unblock* google ads. They're not intrusive, no moving graphics and crap like that, and they're generally relevant, although I rarely click. I'll figure out how to do that at some point.
Tab Mix Plus is great. There are lots of features there, most of which I don't know how to use. The main thing for me is that it has tab memory like Opera has. That is, it's possible to configure it so that when the browser starts, it will auto-load the tabs that were there when the browser was last closed. It also saves tab status while browsing, so that firefox and crashes (rare, but not completely eradicated) don't completely wipe me out. If I've downloaded 15 tabs to view, then tab memory will bring them all back up when I restart firefox. It seems to work better than Opera's memory, but I haven't used Opera in 2 years. I'm sure Opera has made their tab memory much more robust by now, although it was already a great and wonderful feature when I used it back then.
PDF download is convenient since I've never been able to stand acrobat reader displaying itself in the browser and then taking up so long to start. Not that I have acrobat reader on this computer anymore, or any computer I use, but the extra control over PDF downloads is nice. This is a relatively low profit item for me though, I may drop it.
Sunday, March 05, 2006
Work hard, crash
I've been working very hard for about six months now, since I took over management of the development team at work. Previously I'd been a consultant, project based. That was fun. Lots of work, but no management responsibilities.
The company insisted on hiring me on though and after some resistance, I accepted. Things are still fun, particularly when i get to code, but there's too damn much coordination and meeting. All very necessary, but it sure reduces time available for programming.
After six months of this though, reality catches up. I got a very bad sore throat some other stress related infections and just general weakness (due to the combined infections, probably). This happens to me, once in a while. After a sustained period of late nights, not enough sleep, stress, my body crashes for a short period. If I don't get enough of a break, my mind snaps too and I quit ;-). Not going to happen with this job, but it's sure happened before.
Five days of rest have made for an improvement. Not enough though, still going to be weak tomorrow. Damn. I'll rest again on Wednesday.
The company insisted on hiring me on though and after some resistance, I accepted. Things are still fun, particularly when i get to code, but there's too damn much coordination and meeting. All very necessary, but it sure reduces time available for programming.
After six months of this though, reality catches up. I got a very bad sore throat some other stress related infections and just general weakness (due to the combined infections, probably). This happens to me, once in a while. After a sustained period of late nights, not enough sleep, stress, my body crashes for a short period. If I don't get enough of a break, my mind snaps too and I quit ;-). Not going to happen with this job, but it's sure happened before.
Five days of rest have made for an improvement. Not enough though, still going to be weak tomorrow. Damn. I'll rest again on Wednesday.
Tuesday, February 21, 2006
php+postgresql+cursors
I ran into a problem recently because PHP was dying when processing a query that returned some 20 million rows. PHP was running out of memory.
After some googling and running the program several more times (with some runs involving increased memory limit for PHP), it became clear that PHP's postgres support code is loading whole result sets into RAM (and feeding it to PHP one by one, when the PHP code calls pg_fetch_row or pg_fetch_array) rather than using cursors so that it can load one row at a time. I don't know if ADODB or PEAR do better. If they're still using the underlying postgres specific functions though, then they're still going to have the same problem.
The solution is to use cursors explicitly. There are some caveats.
1. Using a cursor requires using a transaction. This isn't onerous, just
something to remember.
2. It's necessary to check after the FETCH to see if there are no more rows.
There's probably a prettier way, but I just go with an infinite while loop
that breaks when there are no more rows.
// assume that $conn is the db connection and that $q is the query, the ff is typed
// straight into blogger, might contain minor errors.
fetching rows one at a time, above, is for illustration only. of course,
for efficiency, one would fetch many rows at a time and loop through them.
After some googling and running the program several more times (with some runs involving increased memory limit for PHP), it became clear that PHP's postgres support code is loading whole result sets into RAM (and feeding it to PHP one by one, when the PHP code calls pg_fetch_row or pg_fetch_array) rather than using cursors so that it can load one row at a time. I don't know if ADODB or PEAR do better. If they're still using the underlying postgres specific functions though, then they're still going to have the same problem.
The solution is to use cursors explicitly. There are some caveats.
1. Using a cursor requires using a transaction. This isn't onerous, just
something to remember.
2. It's necessary to check after the FETCH to see if there are no more rows.
There's probably a prettier way, but I just go with an infinite while loop
that breaks when there are no more rows.
// assume that $conn is the db connection and that $q is the query, the ff is typed
// straight into blogger, might contain minor errors.
pg_query($conn,"BEGIN WORK");
$cres=pg_query($conn,"DECLARE my_cursor CURSOR FOR $q FOR READ ONLY");
if(!$cres)
{
die(pg_last_error($conn);
}
// query succeeded, fetch the rows from the cursor
while(true)
{
$res=pg_query($conn,"FETCH my_cursor");
if(!$res)
die(pg_last_error($conn));
// quit loop if no more rows in cursor
if(pg_num_rows($res)==0)
break;
// there's a row, get it and process it.
$row=pg_fetch_array($res);
process_row($row);
pg_free_result($res);
}
pg_query($conn,"CLOSE my_cursor");
pg_free_result($cres);
pg_query($conn,"END WORK");
fetching rows one at a time, above, is for illustration only. of course,
for efficiency, one would fetch many rows at a time and loop through them.
Saturday, February 18, 2006
Title Wrong - ridiculously
So in my previous post the title says that java studio is ridiculously slow, and then I forgot to mention that fact in the rest of article. On Saturdays I lose my brains.
Anyway, when my colleague was showing me the class diagrams that he developed in Java Studio, I couldn't stand the fact that scrolling across the diagram space, zooming in, and dragging items from one place to another was slow. It wasn't really unusably slow, but it was irritating to anyone used to how fast dragging things in Konqueror (itself slow enough that I never use it, even on a fast box, but a good comparison because, despite it being slow, Java Studio is slower) is.
Maybe things will get better on my own computer though. I'm still waiting for the sun Developer Network site to let me in though, before I can download and test Java Studio Creator. Maybe under strace, in case that can show what's making it slow (I doubt if it'll help, but it's a weekend, I can spend the time looking at strace :-).
Anyway, when my colleague was showing me the class diagrams that he developed in Java Studio, I couldn't stand the fact that scrolling across the diagram space, zooming in, and dragging items from one place to another was slow. It wasn't really unusably slow, but it was irritating to anyone used to how fast dragging things in Konqueror (itself slow enough that I never use it, even on a fast box, but a good comparison because, despite it being slow, Java Studio is slower) is.
Maybe things will get better on my own computer though. I'm still waiting for the sun Developer Network site to let me in though, before I can download and test Java Studio Creator. Maybe under strace, in case that can show what's making it slow (I doubt if it'll help, but it's a weekend, I can spend the time looking at strace :-).
Java Studio - slow - ridiculously
A colleague at work was using Sun's Java Studio Creator for some prototyping. He was running it on the LTSP server and I noticed that it would sometimes eat 100% of the CPU (well, 50% of the CPU since it's a dual-CPU box, the culprit to be mentioned further on doesn't seem to be a multi-thread issue, at least) continuously, while it was doing nothing (everyone had left for home, no one was working in Java Studio anymore).
That seems like a stupidity, and I thought I'd download it to my own computer so I could see if the behavior would manifest there also.
I've subscribed to the Sun Developer's network though, and I can't log in to where I can download Java Studio. They haven't sent me a confirmation email yet, I can't log in. Seems like some slowness over there at Sun. probably due to java. Damn, I'm getting disillusioned with working in Java.
That seems like a stupidity, and I thought I'd download it to my own computer so I could see if the behavior would manifest there also.
I've subscribed to the Sun Developer's network though, and I can't log in to where I can download Java Studio. They haven't sent me a confirmation email yet, I can't log in. Seems like some slowness over there at Sun. probably due to java. Damn, I'm getting disillusioned with working in Java.
Friday, February 17, 2006
First rule of debugging
My wife and I came home from Tagaytay and there was no internet. The cable modem had the receive light blinking and the send and online lights off. OK, I had to get to work and didn't care enough to fix it right then (mistake, Sol wanted to work at home, but it's a short walk to work, so no huge deal). When I got home, the cable modem was still down. I texted the destiny cable help line and went to bed. In the morning, same thing.
I texted destiny again and they called me (but I didn't answer since my phone was on silent+vibrate).
And then I looked at the cable modem and realized that the cable wasn't connected. In removing the laptop power cable for the trip to Tagaytay, I had moved the cable enough that it disconnected from the modem.
I had violated the first rule of debugging, it's *MY* bug. Almost all bugs are my bug. First thing to do is check if I did something wrogn before reporting to anyone else that they have a problem with their system.
No big deal, but useful to be reminded.
I texted destiny again and they called me (but I didn't answer since my phone was on silent+vibrate).
And then I looked at the cable modem and realized that the cable wasn't connected. In removing the laptop power cable for the trip to Tagaytay, I had moved the cable enough that it disconnected from the modem.
I had violated the first rule of debugging, it's *MY* bug. Almost all bugs are my bug. First thing to do is check if I did something wrogn before reporting to anyone else that they have a problem with their system.
No big deal, but useful to be reminded.
Tuesday, February 07, 2006
cpufreq on an AMD 2400+ - strange
I rebuilt my 2.6.12-15mdk kernel, with a Pentium III/Celeron/Coppermine ... CPU processor family because when I built it with either AMD K6 or K7, it was just not showing the right CPU frequencies to choose from in cpufreq.
Or maybe it was, but I didn't like the choices as they all seemed too high.
cat /proc/cpuinfo says (among other, snipped, things):
vendor_id : AuthenticAMD
cpu family : 6
model : 10
model name : mobile AMD Athlon(tm) XP-M 2400+
stepping : 0
Maybe there's a problem with the detection of the CPU frequencies. Or maybe the frequencies it shows are right (unlikely, since the lowest frequence is around 1.3Ghz or so, and the highest is much higher than 2.4Ghz, even though the CPU only goes to around 1.7Ghz in actual clocks.
Anyway, no big deal. I've got a working kernel, it does swsusp, it does cpufreq, it does most of what I need. The only thing I'd really like now would be if I could get the damn ALSA sound working. When I load the ALSA sound driver for this VIA82cxxx, I get no sound, and when I load the OSS driver I can't change the volume in kmix. I can get around that by using headphones with volume control, but I just want the ALSA sound driver to work right :-).
I *did* get it working at some point, or maybe that was with the Mandriva 2005 kernel, 2.6.12 something.
Hmmm, might be time to build a generic kernel now.
Or maybe it was, but I didn't like the choices as they all seemed too high.
cat /proc/cpuinfo says (among other, snipped, things):
vendor_id : AuthenticAMD
cpu family : 6
model : 10
model name : mobile AMD Athlon(tm) XP-M 2400+
stepping : 0
Maybe there's a problem with the detection of the CPU frequencies. Or maybe the frequencies it shows are right (unlikely, since the lowest frequence is around 1.3Ghz or so, and the highest is much higher than 2.4Ghz, even though the CPU only goes to around 1.7Ghz in actual clocks.
Anyway, no big deal. I've got a working kernel, it does swsusp, it does cpufreq, it does most of what I need. The only thing I'd really like now would be if I could get the damn ALSA sound working. When I load the ALSA sound driver for this VIA82cxxx, I get no sound, and when I load the OSS driver I can't change the volume in kmix. I can get around that by using headphones with volume control, but I just want the ALSA sound driver to work right :-).
I *did* get it working at some point, or maybe that was with the Mandriva 2005 kernel, 2.6.12 something.
Hmmm, might be time to build a generic kernel now.
Thursday, February 02, 2006
Kernel rebuilding week
I upgraded my Mandriva installation and some things are not working right, so it's kernel rebuilding week.
I use cpufreq to slow down the CPU when I'm not doing anything CPU intensive. So I needed that to work (/sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies or similar). With the default distro, I get an incorrect frequency list. So I need to build the kernel and tell it what CPU i've got. it looks like the kernel tries to use the powernow K7 module for cpufreq, but /proc/cpuinfo tells me that this CPU is AMD family 6, so I should probably be using the K6 powernow module).
I thought I could use the default distro kernel, but when I do that, the USB keyboard and mouse don't work (very strange). So my first kernel rebuild was just to get usbkbd and usbmouse working correctly. I built a kernel that was pure i386 (for vmplayer, going to test openmosix) and tested with AMD K7, now building one for AMD K6.
There is also a problem with sound. If I use the ALSA driver there is no sound, but if I use the OSS driver, there is sound but I can't change the volume. Not sure what that's about. OSS is fine with me, I don't care very much. There's probably something simple I'm missing there. I just need to figure that out and I'll have sound plus volume control.
Not that that matters very much. the sound on this laptop is ridiculously low in volume, so I'd need to get speakers anyway, and I could control the volume from there :-). But for now, it's a small challenge to either get ALSA working or to get OSS working with volume control. I *did* have ALSA working for a bit there, but I don't remember exactly what I was doing. I might have been playing with a 2.4 kernel for openmosix (but 2.4 doesn't have cpufreq built in and I was too lazy to apply that patch and then fight with openmosix' patch incompatibilities).
One of these days it'll all get done.
I use cpufreq to slow down the CPU when I'm not doing anything CPU intensive. So I needed that to work (/sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies or similar). With the default distro, I get an incorrect frequency list. So I need to build the kernel and tell it what CPU i've got. it looks like the kernel tries to use the powernow K7 module for cpufreq, but /proc/cpuinfo tells me that this CPU is AMD family 6, so I should probably be using the K6 powernow module).
I thought I could use the default distro kernel, but when I do that, the USB keyboard and mouse don't work (very strange). So my first kernel rebuild was just to get usbkbd and usbmouse working correctly. I built a kernel that was pure i386 (for vmplayer, going to test openmosix) and tested with AMD K7, now building one for AMD K6.
There is also a problem with sound. If I use the ALSA driver there is no sound, but if I use the OSS driver, there is sound but I can't change the volume. Not sure what that's about. OSS is fine with me, I don't care very much. There's probably something simple I'm missing there. I just need to figure that out and I'll have sound plus volume control.
Not that that matters very much. the sound on this laptop is ridiculously low in volume, so I'd need to get speakers anyway, and I could control the volume from there :-). But for now, it's a small challenge to either get ALSA working or to get OSS working with volume control. I *did* have ALSA working for a bit there, but I don't remember exactly what I was doing. I might have been playing with a 2.4 kernel for openmosix (but 2.4 doesn't have cpufreq built in and I was too lazy to apply that patch and then fight with openmosix' patch incompatibilities).
One of these days it'll all get done.
Monday, January 30, 2006
Spam mail policies at ISPs
I had some trouble with my internet provider, destiny cable internet last week. Emails being sent out their their mail were not getting sent. I use a local postfix server with the local server sending mail out through the ISP's official smtp server using relayhost. This, because, like most large internet providers, destiny blocks outgoing port 25 (as they should).
I was getting error messages like this:
Jan 24 22:31:25 my-laptop postfix/smtp[4704]: CA13C270A9:
to=, relay=smtp.mydestiny.net[202.8.224.8],
delay=4, status=bounced (host smtp.mydestiny.net[202.8.224.8] said:
554: Client host rejected: Access denied (in
reply to RCPT TO command))
After getting in contact with destiny, I was told that their server though that my IP was spamming and so had decided to block everything from me.
I think this is a mistake. It's a convenient thing to do, notice a few emails outbound that seem spammy and block the IP completely, but it's too prone to error. And users are going to be confused. I'm lucky I have an inside pipeline to the destiny technical people. users who don't have that access would have trouble even figuring out what was going wrong, let alone explaining it so that it could be correctly diagnosed and fixed.
The right thing to do is to check every email going out, and if it's spammy, deny it with a useful error message (e.g., Destiny Internet has determined that this email is spam, please contact technical support if you think it isn't spam [plus some verbiage on how the ISP was protecting the user from spam and apologizing for inconvenience if the spam classifier got it wrong]). That way, emails that are clearly not spam don't get blocked.
Don't block IPs, block individual emails.
I suppose it's possible that that's what their spam classifier was doing. It had seen too much spam from my IP (when the IP was still used by someone else) and the bayesian learning system in there had locked on to the IP so that anything coming from that IP would be spam even if the rest of the email was innocuous. I don't think that's the case though. and it's something they should fix.
To be sure, blocking an IP after a whole bunch of spams have gone out is a cheap way to minimize load on the spamassassin cluster (or whatever they use), but it's still a mistake. Add servers to the cluster if necessary, but don't block an IP (which might have been spamified by someone else and now has been inherited by some innocent subscriber) just because of a history of spamming.
I was getting error messages like this:
Jan 24 22:31:25 my-laptop postfix/smtp[4704]: CA13C270A9:
to=
delay=4, status=bounced (host smtp.mydestiny.net[202.8.224.8] said:
554
reply to RCPT TO command))
After getting in contact with destiny, I was told that their server though that my IP was spamming and so had decided to block everything from me.
I think this is a mistake. It's a convenient thing to do, notice a few emails outbound that seem spammy and block the IP completely, but it's too prone to error. And users are going to be confused. I'm lucky I have an inside pipeline to the destiny technical people. users who don't have that access would have trouble even figuring out what was going wrong, let alone explaining it so that it could be correctly diagnosed and fixed.
The right thing to do is to check every email going out, and if it's spammy, deny it with a useful error message (e.g., Destiny Internet has determined that this email is spam, please contact technical support if you think it isn't spam [plus some verbiage on how the ISP was protecting the user from spam and apologizing for inconvenience if the spam classifier got it wrong]). That way, emails that are clearly not spam don't get blocked.
Don't block IPs, block individual emails.
I suppose it's possible that that's what their spam classifier was doing. It had seen too much spam from my IP (when the IP was still used by someone else) and the bayesian learning system in there had locked on to the IP so that anything coming from that IP would be spam even if the rest of the email was innocuous. I don't think that's the case though. and it's something they should fix.
To be sure, blocking an IP after a whole bunch of spams have gone out is a cheap way to minimize load on the spamassassin cluster (or whatever they use), but it's still a mistake. Add servers to the cluster if necessary, but don't block an IP (which might have been spamified by someone else and now has been inherited by some innocent subscriber) just because of a history of spamming.
openvpn and stalling scp/sftp/rsync
I had a problem with openvpn for a while. rsync, scp would work, they would show progress of uploads/downloads, but then they'd stall at the end, when they should have completed and ended.
They just wouldn't complete the file copy.
A bit of googling (which I could have avoided, since the document was just the man page :-) said to try --mssfix [smaller_number] to tell TCP sessions what packet sizes to use (for the case where MTU discovery doesn't work correctly).
That worked. I tried *far* too small a number, 256. But it works for me, I'm running something over the vpn link and I don't want to cut it off yet since it's very long running. When the processing completes, I'll try larger values for --mssfix. Or maybe not, I don't care about efficiency on that line too much, it's not like I'm downloading or uploading mp3s. Mostly it's just ssh, the occasional svn commit/update, and rsync of my email from work to home :-). That last could take advantage of the efficiency of larger MTU, but it's not important that it run faster, so I don't care much.
Might play with larger values later just for fun though.
They just wouldn't complete the file copy.
A bit of googling (which I could have avoided, since the document was just the man page :-) said to try --mssfix [smaller_number] to tell TCP sessions what packet sizes to use (for the case where MTU discovery doesn't work correctly).
That worked. I tried *far* too small a number, 256. But it works for me, I'm running something over the vpn link and I don't want to cut it off yet since it's very long running. When the processing completes, I'll try larger values for --mssfix. Or maybe not, I don't care about efficiency on that line too much, it's not like I'm downloading or uploading mp3s. Mostly it's just ssh, the occasional svn commit/update, and rsync of my email from work to home :-). That last could take advantage of the efficiency of larger MTU, but it's not important that it run faster, so I don't care much.
Might play with larger values later just for fun though.
Sunday, January 29, 2006
dar - for multi-volume tar-like backups
I just found DAR, Disk ARchive a backup program for linux (and windows and other OSs too, it looks like) that can do slices (i.e., produce multiple files which can fit on whatever removeable media you use). There have been other, similar programs, but this is the first one that I cared enough to actually try to get working :-). Apart from old programs that I wrote myself (mostly for fun, I never actually used them for much) that cut files apart and pasted them back together.
dar -c w2k3 -vv -s 795M -R /[path_to_directory_to_backup]/
there are options for including and excluding files and patterns, etc.
It does differential backups too. Huh, it's time to buy that external DBD-Writer :-). Or just a whole bunch of CD-RWs. I never did buy a DBD-Writer because of the expense (need to use for laptop, and external drives are expensive) and because I don't download movies. Now that I'm playing with lots of vmplayer images though, and also now that my digicam archives are getting ridiculously large, it's time to seriously think about removeable backup (and not just backup to the 40G USB HD).
dar -c w2k3 -vv -s 795M -R /[path_to_directory_to_backup]/
there are options for including and excluding files and patterns, etc.
It does differential backups too. Huh, it's time to buy that external DBD-Writer :-). Or just a whole bunch of CD-RWs. I never did buy a DBD-Writer because of the expense (need to use for laptop, and external drives are expensive) and because I don't download movies. Now that I'm playing with lots of vmplayer images though, and also now that my digicam archives are getting ridiculously large, it's time to seriously think about removeable backup (and not just backup to the 40G USB HD).
Filipino pirates?
I was reading over at jerry pournelle's site when I came across a link to an article on seven who pirated Revenge of the Sith and put it online and went to jail for it.
I wouldn't normally take notice. This stuff is so common (particularly in the Philippines) that even I get upset with all the pr0n that's right out on the street with no one minding. And I don't even have children yet.
But then I read the article and I find that a lot of the names seem to be filipino.
Marc Hoaglin probably isn't, nor is Michael Fousse. And maybe Albert Valente and Ramon G Valdez are hispanic, not necessarily filipino.
But then there are Jessie Lumada, Dwight Wayne Sityar, Joel de Sagun Dimaano, and maybe Stephani Gima.
I wouldn't normally take notice. This stuff is so common (particularly in the Philippines) that even I get upset with all the pr0n that's right out on the street with no one minding. And I don't even have children yet.
But then I read the article and I find that a lot of the names seem to be filipino.
Marc Hoaglin probably isn't, nor is Michael Fousse. And maybe Albert Valente and Ramon G Valdez are hispanic, not necessarily filipino.
But then there are Jessie Lumada, Dwight Wayne Sityar, Joel de Sagun Dimaano, and maybe Stephani Gima.
Saturday, January 28, 2006
clickthecity - damn slow
It's a lazy saturday and I decided to surf over to Click The City to see what movies are showing this week.
The front page loads *very* slowly. Some of that is "Waiting for spots.clickthecity.com", which is their advertising site, I suppose. they need to make that server either much faster, or (more likely), optimize the programs that choose the ads to show. I clicked on a link and now it's been waiting for spots for 2 minutes. Not only are they losing the customer because he closes the window, they're losing me forever because I won't deal with a site that makes me wait that long.
When I got to the movies page, finally, I decided I'd right click on a film and open in another tab so that the information about the film would load slowly in another tab and I could continue looking at the current page. But they probably have some sort of javascript (too lazy to look at the source) that disables right click. They want you to surf their slow site in only one window. Of course that doesn't stop me, I just copy the URL, open a new tab and paste the URL there. but that's just pissing me off, giving me more reason never to visit their slow, stupid and fascist site ever again.
Huh, still waiting for spots.clickthecity.com. That's it, 5-8 minutes waiting for ads is just too much. I'll fix my DNS so that clickthecity.com never ever resolves. Idiots.
The front page loads *very* slowly. Some of that is "Waiting for spots.clickthecity.com", which is their advertising site, I suppose. they need to make that server either much faster, or (more likely), optimize the programs that choose the ads to show. I clicked on a link and now it's been waiting for spots for 2 minutes. Not only are they losing the customer because he closes the window, they're losing me forever because I won't deal with a site that makes me wait that long.
When I got to the movies page, finally, I decided I'd right click on a film and open in another tab so that the information about the film would load slowly in another tab and I could continue looking at the current page. But they probably have some sort of javascript (too lazy to look at the source) that disables right click. They want you to surf their slow site in only one window. Of course that doesn't stop me, I just copy the URL, open a new tab and paste the URL there. but that's just pissing me off, giving me more reason never to visit their slow, stupid and fascist site ever again.
Huh, still waiting for spots.clickthecity.com. That's it, 5-8 minutes waiting for ads is just too much. I'll fix my DNS so that clickthecity.com never ever resolves. Idiots.
Monday, January 16, 2006
Upgrading blues
auto-upgrading in Mandrake/Mandriva with versions 10.1 to higher versions doesn't seem to be dangerous. I've not had trouble with 10.1, 10.2, 2005, 2006 (unless 10.2 *was* 2005, I forget).
so I decided to upgrade in place from 2005 to 2006. Most things worked pretty well. Certainly, there isn't the wholesale breakage that comes with earlier versions where urpmi and rpm themselves break and can't continue at all without some tedious surgery (copying, rpm and all necessary things from a working installation, or installing a fresh copy of the original revision from CD).
Some things need tweaking. This seems almost always to be the case. There was some flakiness in apache (Mandriva uses 2.0 for it's default apache version now, version 1.3 is now apache1), and the php sub-packages are now much finer-grained, so an automatic upgrade didn't upgrade php-postgres support (although I thought it should have). There was more bogosity with apache2 and php 4 versus apache2 and php 5, I was getting some sort of segmentation fault when calling php programs. Since that was probably a throwback from some old apache or php packages that did not get upgraded, I uninstalled everything apache and php related and then reinstalled them (with php 5 now, although in production I use php4, it's a good time to review php5 on my laptop, for future upgrading of production).
As always, the new kernel broke things (sound stopped working in ALSA, although in OSS it usually works (but not always, and I don't know why that is), but then I can't change the volume so I really need ALSA to work, the usb keyboard and mouse don't work if I use the kernel from Mandriva 2005, so I can't stick with that unless I'm on the road). SWSusp doesn't work with this laptop and the original Mandriva 2006 kernel (2.6.12-14mdk), the laptop suspends to disk, but then it won't wake up correctly, instead, freezing. I built a kernel from the 2.6.12-14mdk source that loads from suspend correctly but the sound is still flaky, so I'll probably keep working on this until it works or I give up, download a linus kernel and build that.
so I decided to upgrade in place from 2005 to 2006. Most things worked pretty well. Certainly, there isn't the wholesale breakage that comes with earlier versions where urpmi and rpm themselves break and can't continue at all without some tedious surgery (copying, rpm and all necessary things from a working installation, or installing a fresh copy of the original revision from CD).
Some things need tweaking. This seems almost always to be the case. There was some flakiness in apache (Mandriva uses 2.0 for it's default apache version now, version 1.3 is now apache1), and the php sub-packages are now much finer-grained, so an automatic upgrade didn't upgrade php-postgres support (although I thought it should have). There was more bogosity with apache2 and php 4 versus apache2 and php 5, I was getting some sort of segmentation fault when calling php programs. Since that was probably a throwback from some old apache or php packages that did not get upgraded, I uninstalled everything apache and php related and then reinstalled them (with php 5 now, although in production I use php4, it's a good time to review php5 on my laptop, for future upgrading of production).
As always, the new kernel broke things (sound stopped working in ALSA, although in OSS it usually works (but not always, and I don't know why that is), but then I can't change the volume so I really need ALSA to work, the usb keyboard and mouse don't work if I use the kernel from Mandriva 2005, so I can't stick with that unless I'm on the road). SWSusp doesn't work with this laptop and the original Mandriva 2006 kernel (2.6.12-14mdk), the laptop suspends to disk, but then it won't wake up correctly, instead, freezing. I built a kernel from the 2.6.12-14mdk source that loads from suspend correctly but the sound is still flaky, so I'll probably keep working on this until it works or I give up, download a linus kernel and build that.
Saturday, January 07, 2006
Still dangerous, urpmi update with old Mandriva
So I Tried to upgrade an old (10.0) Mandrake server to 10.2) and it worked for a bit. And then some part of it failed. I did the 10.0 to 10.1, then 10.1 to 10.2 shuffle, but somewhere in the second stage urpmi got trashed and now I can't urpmi (or, in fact, rpm, there are dependencies to URPM.pm and URPM.pm can't be found).
I can recover pretty easily but it's a pain. Of course, I wasn't totally upgrading everything though. That was probably the bug. The last time I tried a complete update using that technique it worked fine. For now I'd say that it's probably possible to totally upgrade a Mandrake 9.x->10.0 and upwards using the multi-stage technique mentioned in the link, but selective updates (particularly selective updates from two generations ago) are a bad idea.
What I did was:
upgrade rpm and urpmi from the versions in 10.0 to 10.1, then upgrade them again to 10.2, but without upgrading everything else. Well, it seems that some packages (e.g., perl-URPM) need to be upgraded but the dependency trees might not be complete, or, at any rate, something goes wrong in the process and some things got lost (URPM.pm, from perl-URPM). they were removed but then installing the new copies didn't work, so now urpmi and even rpm don't work anymore.
Heh, that's the last time I'm doing that :-). Partly, because I don't have any more old servers to maintain, but also because (while it's possible, and not so difficult, to fix it), it's a pain and not worth the time. Just backup, reinstall everything, and restore, is what I say. I couldn't follow that advice with this latest server (won't boot from any newer media, something wrogn in the bios or similar), but I'll do that with everything else.
Of course, at work we're using SuSE and I'm getting ready to upgrade in place with that. Hay, never learn :-).
I can recover pretty easily but it's a pain. Of course, I wasn't totally upgrading everything though. That was probably the bug. The last time I tried a complete update using that technique it worked fine. For now I'd say that it's probably possible to totally upgrade a Mandrake 9.x->10.0 and upwards using the multi-stage technique mentioned in the link, but selective updates (particularly selective updates from two generations ago) are a bad idea.
What I did was:
upgrade rpm and urpmi from the versions in 10.0 to 10.1, then upgrade them again to 10.2, but without upgrading everything else. Well, it seems that some packages (e.g., perl-URPM) need to be upgraded but the dependency trees might not be complete, or, at any rate, something goes wrong in the process and some things got lost (URPM.pm, from perl-URPM). they were removed but then installing the new copies didn't work, so now urpmi and even rpm don't work anymore.
Heh, that's the last time I'm doing that :-). Partly, because I don't have any more old servers to maintain, but also because (while it's possible, and not so difficult, to fix it), it's a pain and not worth the time. Just backup, reinstall everything, and restore, is what I say. I couldn't follow that advice with this latest server (won't boot from any newer media, something wrogn in the bios or similar), but I'll do that with everything else.
Of course, at work we're using SuSE and I'm getting ready to upgrade in place with that. Hay, never learn :-).
Friday, January 06, 2006
Upgrading old Mandrake versions upward with urpmi
Long ago (I can't find the posts on Jijo's archives of the Plug mailing list) I saw a post by Marvin Pascual on upgrading a Mandrake based linux system to the next version by using urpmi. This was great because it wasn't necessary to reinstall the OS,
just pop in the distribution media (or put it on the network somewhere),
urpmi.removemedia -a
urpmi.addmedia [name] [url/filesystem_location] [with hdlist_location] (or similar)
and wait.
upgrades usually too *longer* than a fresh install, and some things always broke, but it was convenient.
when i tried that though, it didn't work for me. in fact it broke the system enough that I had to reinstall anyway (fortunately, since /home was separate, without having to backup and restore my data, although i backed up anyway, so actually i just saved the restore part :-).
I never did figure out why that failed (it was sometime around the 8.x to 9.x upgrade). I tried it again when 10.0 came out. that failed too, as did 10.0 to 10.1. finally though, 10.1 to (i think) 10.2 or Mandriva 2005 worked without error. But I was still stuck with some old servers (from old consulting clients) which were 10.0.
Recently (sometime in 2005) I decided to try something easier, I first downloaded the new RPM and URPMI packages (and all their dependencies) from the newer version, installed those with rpm -i and then, when rpm, urpmi, urpmi.removemedia and urpmi.addmedia had installed correctly, continued with the auto-upgrade from fresh media. That worked. So I had found a way out of the point where the process had blocked. Apparently, there was something wrong with how it was downloading one or another of those files (and their dependencies, might just have been some order of installation issue, or perl installation order errors too, I don't remember).
So now it's possible to upgrade 8.x to 9.x, 9.x to 10.0 and 10.0 to 10.1 (10.1 upwards are fine without this workaround). I thought I wouldn't need this information anymore, except a year and a half ago or so I completed installation on a server that, for some reason, would not boot from the 10.1 DVD. So I had to install 10.0 from CDs. I need to upgrade some packages on that box though, and install some other packages that weren't available (or were seriously deficient) in 10.0.
So I followed the procedure I've outlined:
urpmi.removemedia -a
urpmi.addmedia [the new media, actually, i just used web sources for the
packages since the server is for an ISP and their bandwidth is huge]
urpmi rpm (say yes to the dependencies)
urpmi urpmi (again, say yes)
then urpmi anything else necessary. i haven't upgraded *everything*, but if something breaks, well, i'll urpmi anything that might need it.
Kind of useless knowledge now (can't think of any other servers that i've installed that are older than mandrake 10.1), but there it is anyhow. maybe an old client will contact me about this. or maybe it'll help someone.
just pop in the distribution media (or put it on the network somewhere),
urpmi.removemedia -a
urpmi.addmedia [name] [url/filesystem_location] [with hdlist_location] (or similar)
and wait.
upgrades usually too *longer* than a fresh install, and some things always broke, but it was convenient.
when i tried that though, it didn't work for me. in fact it broke the system enough that I had to reinstall anyway (fortunately, since /home was separate, without having to backup and restore my data, although i backed up anyway, so actually i just saved the restore part :-).
I never did figure out why that failed (it was sometime around the 8.x to 9.x upgrade). I tried it again when 10.0 came out. that failed too, as did 10.0 to 10.1. finally though, 10.1 to (i think) 10.2 or Mandriva 2005 worked without error. But I was still stuck with some old servers (from old consulting clients) which were 10.0.
Recently (sometime in 2005) I decided to try something easier, I first downloaded the new RPM and URPMI packages (and all their dependencies) from the newer version, installed those with rpm -i and then, when rpm, urpmi, urpmi.removemedia and urpmi.addmedia had installed correctly, continued with the auto-upgrade from fresh media. That worked. So I had found a way out of the point where the process had blocked. Apparently, there was something wrong with how it was downloading one or another of those files (and their dependencies, might just have been some order of installation issue, or perl installation order errors too, I don't remember).
So now it's possible to upgrade 8.x to 9.x, 9.x to 10.0 and 10.0 to 10.1 (10.1 upwards are fine without this workaround). I thought I wouldn't need this information anymore, except a year and a half ago or so I completed installation on a server that, for some reason, would not boot from the 10.1 DVD. So I had to install 10.0 from CDs. I need to upgrade some packages on that box though, and install some other packages that weren't available (or were seriously deficient) in 10.0.
So I followed the procedure I've outlined:
urpmi.removemedia -a
urpmi.addmedia [the new media, actually, i just used web sources for the
packages since the server is for an ISP and their bandwidth is huge]
urpmi rpm (say yes to the dependencies)
urpmi urpmi (again, say yes)
then urpmi anything else necessary. i haven't upgraded *everything*, but if something breaks, well, i'll urpmi anything that might need it.
Kind of useless knowledge now (can't think of any other servers that i've installed that are older than mandrake 10.1), but there it is anyhow. maybe an old client will contact me about this. or maybe it'll help someone.
Sunday, December 25, 2005
Suse (even opensuse) on Vmware
I use Mandriva as my preferred distribution. The *drak* stuff isn't that big a deal for me, I prefer the command line, usually, and it still isn't easy to add an ethernet interface alias via drakconf (actually, I don't think I see how to do it at all, although you can do it with webmin). I'm *really* in love with the urpmi stuff though (and mainly the command line urpmi, although rpmdrake is very good too).
I thought I'd try suse though, either the evaluation version, or opensuse. But I can't afford to overwrite my laptop since I don't want to have to reconfigure everything (openvpn to two different networks, etc). So I thought I'd try it with vmware. When I downloaded the DVD ISO, it wouldn't work in vmware. It kept giving some sort of boot error. I thought maybe the DVD download was corrupted so I downloaded again, same problem. Next I downloaded the CDs. Same problem. It looks like opensuse (or the suse evaluation version) just won't install in vmware. That's a shame, since I'm certainly not going to switch to it if I can't test drive it, and I just don't have a spare computer to test drive it on. Ah well.
I thought I'd try suse though, either the evaluation version, or opensuse. But I can't afford to overwrite my laptop since I don't want to have to reconfigure everything (openvpn to two different networks, etc). So I thought I'd try it with vmware. When I downloaded the DVD ISO, it wouldn't work in vmware. It kept giving some sort of boot error. I thought maybe the DVD download was corrupted so I downloaded again, same problem. Next I downloaded the CDs. Same problem. It looks like opensuse (or the suse evaluation version) just won't install in vmware. That's a shame, since I'm certainly not going to switch to it if I can't test drive it, and I just don't have a spare computer to test drive it on. Ah well.
Subscribe to:
Posts (Atom)