For the record, htdig can index through the local filesystem (bypassing the HTTP protocol.) You are correct about htdig not supporting multi-byte characters.
