There's not much you can do about it except to ban each of the offending IP.
These are probably scripts/programs that try to automatically submit (spam) thousands of guestbooks/forms/forums - thanks to the GBook protection they don't come through, but they are trying anyway. In last version of GBook I added IP banning based on certain criteria (if it's 100% spam then the IP is banned) and I already have over 6700 IP's banned so the numbers are huge.
I use AwStats and what I do once per month is check who the most frequent visitors (IPs) are, do a little research (IP whois lookup, see what pages they've been to etc) and I end up banning most of them (or their IP ranges) directly in the server's firewall. It's quite some work and tricky work that is (you need to be carful not to ban Googlebot for example).
I don't think your bandwidth costs are going thru the roof just because of scripts trying to spam GBook though. Here's an greatly exaggerated example:
- 15Kb for the Gbook HTML
- 65Kb is ALL of the images (smileys) are loaded on page
This makes 80Kb, let's say it's 100Kb.
Let's say scripts try to spam your GBook 10.000 times per month (another hughe exaggeration), this would bring you to about 1Gb traffic per month.
Note the above is an exaggerated estimation, but even if it was true 1Gb traffic costs $0.5-$2.0 and that isn't thru the roof

It is very annoying though, I agree.
So, unless you can ban (at least the large spammers) from the server I'm afraid there isn't much you can do about it. I already accepted it as a part of today's online business. And as long as their SPAM is blocked I'm happy.