Large copies to USB 1.0

I've got a laptop that only has USB 1.0. This is great for my USB keyboard and mouse (there are no PS/2 ports at all, so external input devices *have* to be USB, I've never heard about PCMCIA keyboards or mice :-) and also for my USB bluetooth and memory sticks, but it's terrible for a USB hard drive.

Recently I was copying multi gigabyte files to the USB drive and I couldn't stand that it was taking so long and that the laptop would pause and block for tens of seconds. I thought what was happening was that the data was being buffered and then, when the data was actually written down to the hard drive the kernel was so busy handling interrupts (or polling, or whatever) that the computer blocked.

I came up with two workarounds. The first workaround was to set the buffer flushing period to very short, so that the laptop wouldn't block completely for tens of seconds but instead would just get slow every few seconds.

update -3 32 -f 1 -s 1

The second workaround was to limit the bandwidth going down the USB pipe. Rsync is a great tool. This time I couldn't use it to do incremental syncing since in the configuration given checksumming would be far slower than just copying the data file over directly. I didn't want to fill the pipe though since I thought I'd noticed the laptop choking a bit even with the update trick. So what I did was use rsync as a slow cp. Rsync has a --bwlimit option for app level bandwidth limiting which works even within the same computer. So:

rsync -a --bwlimit=384 [srcfile] [dstfile]

limits the amount of data transferred to 384 kilobytes per second. The link can handle up to around 1 megabit per second, but limiting the copy bandwidth keeps the laptop usable and then I just hid the rsync window so that I could continue using the laptop.

rsync --bwlimit wouldn't be enough here. Without the update trick (making buffer flushes shorter and smaller) the buffers would still get filled with hundreds of megabytes of data which would then take around one second per megabyte to go down the wire. With update and no --bwlimit, I was finding slowness and inconvenience. Together though, they made the laptop usable while it was copying large files in the background.

Eventually though, I'll just have to use a laptop that's got USB 2.0 :-).

tiger

600+ kBps!

I downloaded OpenOffice.org 2.0 just now and I was shocked at the download speed. I downloaded through bittorrent and I was getting 600+ kilobytes per second. that's one advantage of going with an ISP that doesn't do bandwidth capping. I use destiny cable internet and while I've never before seen 600+ kBps, i regularly see 100-200 kBps.

I guess someone else in the destiny network was seeding the torrent. Or maybe there's just a whole lot of bandwidth available on monday morning and I got all the benefit of it.

I downloaded the SUSE 10 eval DVD ISO a week ago and that came down at around 60kBps on average. There's something to be said about downloading an ISO via the official torrent too. When SUSE 10 was very new I tried to download with the official torrent and that was so slow (2-3kilo*bits* per second) i decided to download it over the edonkey network instead. That download completed but the ISO was bad, so it was quite a waste of time. When i downloaded from the official torrent again, it slowly went up to around 60 kiloBytes per second. So official torrent+wait+a+few+days is the right thing to do I guess.

Possibly, too, downloading other (more popular)distributions would be better via the official torrents. I've done my bit at misinforming people about SUSE licensing (old information from when it was still SUSE and not novell). Probably people stay away from SUSE because they still have the impression (as I did, until I was gently corrected by a SUSE/novell representative) that SUSE is not yet freely distributable. Well, the eval edition *is* now freely distributable.

Anyway, I'm waiting for Mandriva 2006 to become available to see what the download rate on that will be :-). I'm the second worst type of ISP client, really, downloading things sometimes only to see what the ISP's performance is like. The worst type, of course, are those guys with huge hard disks who download movies, games and mp3s as if there were no tomorrow.

tiger

rsync algorithm talk by tridge

Andrew Tridgell has a great talk on The Rsync Algorithm

rsync and --partial

rsync --partial doesn't do what I thought it would do. In fact, for large files (my current project involves rsync of half gig files over relatively slow networks, and often networks which are relatively unstable) it can actually make the destination file lose so much information if the rsync dies early that it would become necessary to re-download practically the whole file.

I was under the impression that --partial did something smart, e.g., maybe it would be truly incremental. That is, if a large file was to be downloaded (from source to dest) but the download died somewhere in the middle of the file, that all the *good* data from the source half that was correctly downloaded would be saved AND THEN the other half (on the dest) would be *APPENDED* to the partly downloaded half. This way, if the download were to be continued, then the already good half would all hash correctly and the download would start at the first wrong block.

Or maybe --partial would keep all of the good downloaded file in those temporary .[real_filename].[random_letters_and_digits] files and the temporary would not be removed so that the next download would use the same temporary file.

Unfortunately, --partial doesn't do either of those. It's pretty good if you're downloading a completely new file (the destination didn't exist before or it's *much* smaller than the original). In that case, the file is partially downloaded and when the rsync dies, the partially downloaded file is kept. If you tried to do this with a large file already downloaded, and the new download died before it got to the size of the already downloaded file, you'd *lose* data since the short newly downloaded file would *replace* the old, larger, already downloaded partial file.

So --partial is very useful when downloading files from scratch, but it's not useful (in fact, may be very harmful) when syncing files which you've already partly downloaded before. For that, just use the regular rsync parameters and don't use --partial.

As good as rsync is, I think there's quite a bit of improvement yet to be done for the case where the bandwidth is slow/unstable and the files to be transferred are very large. And either of the suggested mechanisms above might be a good thing to implement.

Resumes with no email address

I received a resume a few weeks ago and the candidate is very interesting. It's in the U.S. style, two pages with only relevant information in there. the philippine style, with parent's names, religion, and picture, is useful for discriminating against people based on surface characteristics. I discriminate against people based on stupidity, and while that can sometimes be seen in resumes, more often I have to wait until the interview for that.

Many of his skills are just what we need, and he's interested in learning other skills that we need that he doesn't quite have yet. That he studied at Mindanao State University and Ateneo de Davao and graduated from the University of the Philippines at Diliman doesn't hurt either.

But his resume doesn't have an email address. Or a celphone number either. Maybe HR got this from JobsDB or similar and they need to pay extra money to get his contact information. Whatever the case, I dropped the resume and the positions have since been filled.

Resumes with no convenient contact information is just not going to be seen. Well, by me, anyway. He's got a snailmail address there too. But it's in Los Angeles, CA, for some reason. Now that's another filter. If he's in the U.S., he's either immigrating (legally or otherwise) or he's just visiting. If the first, why submit a resume to a philippine based company? If the second, he should put down his philippine snail address since a U.S. address is instantly off-putting. Why waste time considering someone who is probably already making more money in the U.S. as a busboy than he would make in the Philippines?

Of course there are exceptions. There are people who go to the U.S. for a few months, work there as computer consultants or software developers for very small (t<6 months) projects and come home. But there should still always be an email address or at least a chikka-able number, so that the employer can check whether any such exceptions apply.

Finally, bluetooth on Linux

I've finally got Bluetooth working in linux. For a while I was just looking at bluez and I couldn't figure out how I was actually supposed to transfer files. I could connect, the pin was right, but I didn't know how to send or receive files.

I installed gnome-bluetooth-manager and I still couldn't figure it out. All that did was see the phone. I couldn't do anything to the phone connection or find a way to transfer files.

Finally though the kdebluetooth RPMs finally made it to my urpmi server and after i did urpmi kdebluetooth I'm now able to transfer files. It took a while to figure out which programs to use (didn't realize that they'd be in the icewm menus too, I'm a moron). But finally found kbtobexclient and was able to transfer a demo java game (J2ME with the wireless toolkit makes development easier, not easy [yet, but i haven't worked on it much yet except to look at the demos], but easier) and actually run it.

Moving Tech blog entries here

My main blog is at Bounding Overwatch. I've created this new blog as my tech blog though (struggling very hard to avoid using some amusing yet faux latin for the blog name) and I've copied most of the tech related blog entries from the old blog to here.

The entries are still there too though. Didn't want links to break if people are linking to me there or search through google.

I wonder what the result of all this moving will be for jijo though. His Filipino Tech blog aggregator picks up posts via RSS. But if I posted all of these entries this morning (except I used the original datetimes), will his aggregator be suddenly inundated with all my entries? Or will only this first entry make it?

It's probably the second, but it'll be interesting if it's the first. Hahaha.

Dynamic DNS in Linux

That title is a bit over the top, of course. I only looked at one solution, and when that worked for me, I stopped looking.

I've got a sort of static IP. My internet provider, Destiny Cable Internet, as of this writing, uses the ISC DHCP Server. That server tries to allocate the same IP to the same requesting MAC if the IP is still available. So the IP is pretty stable. However, there is no promise that the IP will never change, so it's semi-stable.

I thought I'd be OK with just assuming that it was stable, but I changed my mind. It *might* change and when it does, then I'll have to go to everywhere that assumes it's stable and fix it. And then it might change again.

Instead I decided to use a dynamic DNS server. For the client software, I first looked at EZ IP-Update and, since there's a list of dynamic dns servers at that site, I chose DynDNS. Both choices were pretty random, but I'm happy with both of them.

ez-ipupdate took around 20 minutes to set up (there was a minor compile error, something wrong in the ./configure script's deciding how to use errno, so I had to figure that out and fix it, it's a quick and dirty hack though, not good enough to propagate upward, but I may email the ez-ipupdate maintainer so that he can look at his configure configuration) and DynDns took 2 minutes (plus the less than a minute to wait for the email to arrive).

So I've got dynamic dns working now. bopolissimus.homelinux.net

Port forwarding with ssh

I find forwarding ports (and, actually, also reverse forwarding ports) over ssh very useful. I'll probably find it less useful now since I've found OpenVPN so easy to setup and use, but it's still a very useful technique, particularly where I don't have root access.

I always use -v when doing this because -v will tell me if the link is slow or the link is down (while i'm still connecting, less wasted time). It also shows a message everytime a request is forwarded down the ssh tunnel, useful for debugging to make sure I'm doing the right thing.

With the setups below, surfing to http://localhost:8080 will forward requests to the internal or external server specified. Note: by default, port forwarding binds only to 127.0.0.1, so from the box itself you can connect to the forwarded service, but you can't from another box. Use GatewayPorts for exposing the service (warning, think about that, it might open security holes).

1. I'm on my laptop and I want to surf the website on remote.com.ph as if I were physically there. So what I want to do is forward my port 8080 to remote.com.ph:80 (it's an internal website and is inside a firewall, so I can't surf to that site directly from the open internet).

ssh -v -L 8080:localhost:80 remote.com.ph

what that does is, connections on my local port 8080 will be forwarded to remote.com.ph. From there, it will be forwarded to remote.com.ph's localhost:80.

2. Same as #1, I can ssh to remote.com.ph but the web server isn't on remote.com.ph, it's on another internal box inside the firewall, 192.168.80.80:80.

ssh -v -L 8080:192.168.80.80:80 remote.com.ph

What that does is, connections on my local port 8080 will be forwarded to remote.com.ph over the ssh tunnel. Remote.com.ph will then forward them onward to 192.168.80.80:80

3. Same as #2, but I'm trying to surf to some external web site that doesn't let me in if I surf from the Philippines, or if I surf there from the Philippines something bad will happen. For instance, PayPal will block paypal accounts if they're used from IPs that it identifies as being in the Philippines. This has to do with very high fraud rates. But maybe I'm not a fraudster, I just want to use my paypal account, but I can't because I'm physically in the Philippines. If I have ssh access to a host in the USA which paypal won't be suspicious about, I can do port forwarding through that server, e.g.,

ssh -v -L 8080:www.paypal.com:80 my_us_server.com

There is also reverse port forwarding. I do that when the server I need to connect to (usually not for http, but for some internal server) is inside a firewall and I can't get to it directly and the firewall won't port forward to the internal server. In that case, what I do is ssh to the gateway, and from there ssh to the internal server. Then I do reverse port forwarding. What that does is, it will ssh back to *me*, and open a localport on *me* which will be forwarded down that second ssh link back to it. This, of course, only works if the internal server has a route out to the internet, if it doesn't, then some other solution will have to be found.

# first ssh to the gateway
ssh gateway.remote.com.ph

# at the gateway, ssh to the internal box
ssh my-internal

# at my-internal, open the reverse tunnel back to me, i am, client.com.ph

ssh -v -R 8022:localhost:22 client.com.ph

What that does is get my-internal to ssh to client.com.ph (my box, outside the firewall). Once it gets there, it will set up port forwarding so that port 8022 at client.com.ph so that when I (at client.com.ph) connect to port 8022, the request will be forwarded to port 22 at the my-internal computer. I use that much less often than -L, but when -L doesn't work, -R is often a lifesaver.

Comparisons of different SQL implementations

Online there's a pretty long (not sure how comprehensive, but informative and useful) Comparison of different SQL Implementations

Haven't read it all yet (busy), but I'll need to get to that within 1-2 days.

Destiny Cable Internet SMS number

To get Destiny Cable Internet to call me (in case of technical problems),SMS to 09175771111 MD [account name] [account phone number]

Very convenient since I have a cell phone and cell phone charges being exorbitant in the Philippines, I'd rather not have to call them. Besides, I don't know what the tech support number is and I'm too lazy to look :-).

It's weird that one has to put one's phone number in there, but I guess that's for people who have landlines. They can put their landline number there. I don't have a landline so it seems a waste to type the number in since they should just call me on my cel and they've already got that when they receive my text.

It's nice to be able to add (after the phone number) some text about the particular problem. So they can look at things on their side (e.g., signal level, problems in my area, etc) before even calling me. Saves me time explaining things, and saves them some telco costs.

Windows Solutions

Long ago I saw a list of "Windows Solutions". I had it in my sigs and now I can't find them. Fortunately, I keep my old sig file around even though I don't use it anymore (I haven't seen a tool yet that will modify the gmail sig via cron).

Anyway, here they are, so I don't forget them, hopefully google will cache them and google can find them with "Windows Solution bopolissimus" :

1. Don't use Windows
2. Stop the program and start it again
3. Ctrl-Alt-Del and kill the program in Task Manager
4. Reboot
5. Uninstall the program and reinstall it.
6. Reinstall Windows and Everything else
7. Backup, format the disk, install windows, reinstall everything else, restore your data.

to be fair, you sometimes need to do all of these (or the transliteration) for other operating systems, and #2, #3, #4 aren't so common anymore in XP. But they're still much more common there than in Linux. All that spaghetti inside is turning into much and Windows is turning more and more into a big pile of mud (if I may be allowed to mix metaphors)

Charles Murray on The Inequality Taboo

http://www.commentarymagazine.com/production/files/murray0905.html

GoogleTalk

I was going to whine about GoogleTalk invites that I got. Three people invited me and when I read the invite it says that the client is windows only.

But on the GoogleTalk site there are indications that the protocol used is Jabber. And in the Help Center link (they really should put this information on the main site) there's a link to How to install GoogleTalk on Gaim. So I'm happy. I don't have the voice chat features of GoogleTalk. I wish I did. I'll probably figure that out at some point. Particularly since I've now got lots of bandwidth, enough for voice chat or video. But I'll do that some months in the future. I don't have a headset and mic setup and I just don't have the time to deal with that right now.

Zotob and why the victims deserved it

I didn't know anything about Zotobuntil days after it had been out. Well, two days after it came out. Or maybe three.

I have the same reaction as every other time something hits the Microsoft side of the world. Everyone who was cracked deserved it. If anyone was injured because a Microsoft controlled warship misbehaved, or maybe some sort of system in space stations, orbiters, space shuttles or hospitals injured or killed someone, well, the people injured weren't to blame. However, the armed forces, space or hospital agencies (and all those financial and banking agencies and power generating and distribution agencies) that chose to deploy and continue to use Microsoft software for anything mission critical are certainly to blame.

They should have patched.

They should have bit the bullet and spent the millions necessary to upgrade to the most current version (XP), and then they should have patched, once, twice, and three times. There would still be holes in there, but they'd not have fallen to Zotob (if they were still at W2K) or any of the various XP remote root attacks.

They could also have switched to Linux. I think that there's a lot of value in Linux for common office workers. They won't get windows viruses, they probably won't get any Linux remote compromises (since they're desktop users and most remote compromises are to commonly run daemons or to web based applications, which regular desktop users just don't run, and even if they got compromised, most compromises on web programs can't promote themselves to root, so they're limited to web page defacements which are embarrassing [to professional linux administrators] but probably unnoticed among regular users, if they even run web servers]).

There's a lot of value in Linux for server applications too, but those need to be locked down more seriously than do desktop systems. Although, generally, they're far less insecure than comparable windows systems. But this isn't an anti-windows rant. It's more a rant against unthinking windows administrators who aren't paranoid about their systems. Any administrator should always be paranoid. But windows administrators should be certified insanely paranoid. They'll get cracked anyway, but at least they'll do their best. And they'll get cracked less often. The amateurs who let Zotob into their systems should be fired and then shot as a favor to the gene pool.

Chikka on gaim needs gaim-devel

There's been a Chikka plugin for Gaim for a while now. I got it to work some time ago, then reinstalled linux and never found the energy or need to get it to work again. It's pretty easy, but I got sidetracked by a desire to make it work with source built gaim rather than using the gaim that already came with my distribution.

I decided to look at it again from the standpoint of just getting it working instead of trying to make it smarter (so that it could detect where gaim was installed if it wasn't in the standard places, etc). When I last did that, I lost interest again since it didn't work.

Today I tried again and I realized that I probably needed to install gaim-devel to get chikka working. and after a:

urpmi gaim-devel

and a simple ./configure;make;make_install in both the chix and chix_gaim directories, i've got chikka functionality in gaim. Now, I don't think it's working yet. I can go online in chikka and i see when other people are online (they just received or sent a chikka message), but messages i send don't seem to be working. I'll need to look at this again some more. I'll probably get it working tonight.

anyway, installing chikka does need gaim-devel. Without it, i couldn't even get gaim to notice that it was possible to connect to the chikka network, or that chikka even existed.

Incredible Speed with Destiny Cable Internet

I was testing downloads overnight and I was getting speeds in the range of 200+ kBytes per sec, which is around 1.5mbps. I didn't think that was possible since I thought the modems themselves were capped at some level (even if destiny internet doesn't have a network appliance doing bandwidth capping per individual user). But I guess the modems aren't capped. I think I did hear mention of 1.5mbps being the physical limit though. Maybe the modems aren't capable of any more than that. But I'm going to test some more to see what the real limits are :-).

Boy, this is great.

Destiny Cable Internet

I need Internet access at home since, as a consultant, I often work offsite and while the QSR offices are close enough to walk, sometimes I just need to work past midnight or on weekends. I can still get to QSR on weekends, but it's just nice to be able to work from home.

I considered DSL (mainly, PLDT MyDSL), but i decided on Destiny Cable Internet due to price and the fact that PLDT MyDSL is capped at 384kbps while Destiny has no caps. Of course it's cable though, so the bandwidth is shared. Destiny just doesn't do any bandwidth capping at all. The management thinks that the thing to do is just to provide the bandwidth (their bandwidth is greater than they need,that is, they buy more bandwidth than actually distribute). I like that attitude, even though it might be better for me (or for the generic customer) to have a cap so that bandwidth hogs don't slow down everyone connected to the same node.

The service has been pretty good. I'm getting consistent downloads in the 400-600 kbps range. I haven't really been monitoring very much yet, or consistently enough. But I'm waiting to get to 1Mbps. QSR has destiny bandwidth too and once or twice I've seen downloads in the megabit range.

Of course, to people from the U.S. (all my brothers and my sister have been through here lately to be with my stepfather before his heart surgery) that's not much. 1.5mbps at home is pretty common there, if you're close enough to a telco switch. But 400-600kbps is great here. and i'm confident that Destiny will continue working on their line quality and will eventually keep me in the megabit range continuously.

Not that I can do all that much with that bandwidth. But it'd be great to have it when necessary.

at-spi really slows down evolution

I couldn't stand how slow evolution was anymore. I know that I'd fixed that problem long ago (with some sort of gconf-editor, ah, edit). I remember that it was some sort of accessibility thing.

Well, I reinstalled linux a while ago and I got evolution slowness again. I've finally found out what I need to do to remove the slowness.

urpme at-spi

Back when I was still installing mandriva, I thought I'd try out the accessibility stuff to see if mandriva was good enough to use for blind people. I still don't know since, well, I can't get gnopernicus working on this laptop. Maybe I can try it out on a desktop. If it's usable then, I may try to get my brother in law (great programmer, and blind) to try out linux.

For any blind programmer, I think the thing to do is to work in linux since the command line development environment and demand for command line developers is high in linux. In windows, well, he programs in VB, and he's pretty good at it, but he's always going to be hobbled by the
graphical emphasis of VB and the command line weakness of Windows. It's probably a great environment for blind computer *users*, but it's a terrible development environment for a blind programmer.

vim settings i always use

set ruler
set noincsearch
set nohlsearch
syntax off
set ts=4
set ai

i should probably also use
set ic

ok, that's it, i will :-)