[fetchmail] Fetchmail from trusted hosts w/o SSH-Agent?

Hi, I am trying to port my existing fetchmail configuration from one debian 
box to another, and I've realized one potential stumbling block. 

To date I've used fetchmail on a local machine, and I run SSH-agent such 
that whenever X starts up, I enter my passphrase once, and fetchmail can do 
its work unassisted. 

But now I'm setting fetchmail up on a remote server that I ssh into.  I want 
fetchmail to be able to fetch my mail from my mail server to the remote 
machine, unassisted.  But it strikes me that may be difficult short of 
SSH-agent, which I can't run since I am not running X on the remote machine. 
Any advice? 

Thanks
Todd