A co-worker was raving about cURL. It looks pretty powerful as a general web-getter toolset and library. Libraries appear to be available in the language of your choice. Java for me of course. But ya never know.

There is also WebSPHINX, an alternative that’s been around for a while.

I’ve used Jakarta’s HTTP Client in a project… but the documentation made it real difficult.