I am recently out of college and two friends and I have decided started creating a system for a law firm. One of our objectives is to create a file type for this system that is encrypted until a password is entered when accessed outside of the system(ie if they want to use Word to view the file). Where can I find resources on how to create my own custome file type? We are using javascript, ajax, and php(this is an web app obviously).

What you want to do isn't going to work without a LOT of effort on your part to write plugins for every relevant application and operating system to read your files.

File formats can be anything you want them to be. What they look like and how they're defined is all up to you.
Write a document defining the format, write some software to read and write that format, and hope people start using it (but don't hold your breath).

K...obviously we could just make the files safety the users responsibility when it is accessed outside of our system. Now that i think of it that is kind of the way it should be in the first place

At least until an intern gets a tape drive(which they never should've had) stolen out of their car, then what?

if you rely on a custom file format and encryption you're not going to be much more secure (and probably less secure, as people will become lax more easily in a false sense of security) than when you implement proper data security procedures and enforce them.

If you want security, contract a dedicated firm to set it up for you and help enforce it.