If you're relying on the HTTP referrer header to prevent hot linking there are a couple of issues you might need to think about. The header can be spoofed. And it's not uncommon for the referrer to be blank, such as when someone bookmarks a resource.
I haven't attempted to block hot linking myself, but what I would try doing is setting a domain cookie so that at least you know they've visited your site. Then when they request the download, their browser will include the cookie in the request header, which you can test against.
If you need to protect resources more thoroughly, consider implementing a way for users to authenticate themselves, such as with a username and password, and/or restricting access by IP address.