I want to protect my site from hacking. Currently I know about XSS and SQL injection.

Do I need to use mysqli instead of mysql? And why?

When should I use htmlentities() and striptags()?

I also don't want users to upload melicious files and since I accept file uploading, is it enough to check file type? If not what can I do to prevent this?

My website runs on PHP, is there anything else I should worry about?

I'm afraid this is a huge list that would take a huge amount of work to explain. Perhaps you should ask a series of simple questions, all with suitable titles, so that they can be searched and accessed easily by others.

Can you at least answer the question? I want to learn the basics first.

Which question? You ask about 4 or 5. That was my point.

As alan.davies wrote you you made a lot (not just four) questions and answering in detail those would take a lot of space , not mentioning explaining those answers. I will give you some (incomplete) quick answers and I hope to use them in order to search and learn more.

//Do I need to use mysqli instead of mysql? And why?
You must use prepared statements. In PHP is either mysqli or (my choise) PDO. Prepared statements tell the DBMS what the parameters that will be sent will used for , so the DBMS (MySql in your case) knows how to protect itself from SQL injections and more. The why here is obvious , because I think you don't want a kid playing (yes kids nowadays do sql injections while learning) bring down your site or even worse access (or delete) your db.

/​When should I use htmlentities() and striptags()?
Almost never. The exception is if you care about a UI injection (this to be serious can be done in numerous other ways). But if you want to avoid it use htmlentities before saving the field.

//I also don't want users to upload melicious files and since I accept file uploading, is it enough to check file type? If not what can I do to prevent this?
Never upload a file in public_html / www or whatever your web root is called. Use another folder (e.g. a repository) that is above the web root. Then use a class (PHP file or whatever) to serve that. In that way you can serve it only to people that have the authority to inspect it. If we are talking about images , never save the original but what you parsed with imagick or similar extensions (there are libraries also for videos).

//My website runs on PHP, is there anything else I should worry about?
Hundreds if not thousands issues.

Thanks a lot @jkon for this helpful information. I will need to look more into it.