I want to work on an antivirus software program.
But, what should I know first to dig?
You need to study how viruses work first, then look at how existing antivirus programs work. Its not easy, and needs a really deep understanding of the operating system.
In any case, Java is completely the wrong tool for this job. It's designed to be independent of any one operating system, and provides a protected enclosed vistual machine for your program. Your anitvirus will need to dig deep into the running operating system and file systems to look for hidden viruses. This is one of those rare cases where C (or maybe C++) is almost certainly your best option.
First, you shall NEVER even consider making a malware removal application in Java. Java is not good and is not meant to be used to create such software. The reason for this is, that Java can't "dig down" into the system like more low level programming languages can. Instead programming languages such as, ASM, C and C++ would be elementary to create such tool.
Second, you haven't mentioned which operating system you want to create this tool for? Since the majority of people in here uses Windows I will answer from the perspective that you uses it too.
Before even starting on the process of programming, you should look into how malware behaves and where it hides. You should get really comfortable with the Windows Registry and know about every single part of the operating system. Creating a tool to remove malware is a REALLY and I mean REALLY big process.
Thanks for your time to reply and providing useful terms.
Can you provide me some links for further readings?
I'vce never got into anti-virus programming, so I'm sorry, you'll have to Google it yourself.
Given your initial question, you should probably give up trying to write any malware removal software as if you'd have even the most rudimentary knowledge about such you'd have known Java is not an appropriate choice to implement it.
That's a fair point jwenting makes. You may find it easier to take up cardiac surgery or training fighter pilots . No, I'm not joking. Antivirus really is that hard.