I found the solution at ubuntuforums:
http://ubuntuforums.org/showthread.php?t=1314958
I do run screen on the remote desktop (my work desktop), and I generally start screen when X is already up, so *that* has the correct $XAUTHORITY entries. So I can just connect to the running screen instance and do:
echo $XAUTHORITY
That will have something (from the ubuntuforums link above):
/var/run/gdm/auth-for-fred-s0G6r1/database
and, from ssh, I can use that:
x11vnc -display :0 -auth /var/run/gdm/auth-for-fred-s0G6r1/database
I should probably just have .bashrc detect if there's a $XAUTHORITY variable and if there is, echo it out to a file that I can then just source :-).