I want to know is it possible to do that and install your own custom software on it ?
Possible? Yes. There are folk that create new "roms" or images all the time at XDA Forums. But the toolset and experience is something that won't fix in just one book.
You may have to rub shoulders with that now defunct Firefox OS on a cell phone effort. The hardware looked no different than a run of the mill Android phone.
You have to flash the ROM and reload the OS.
I see you wanted a little more.
Flashing most Android devices is well, done with tools like http://donandroid.com/odin-tutorial-how-to-use-odin-to-flash-rom-on-android-device-634 BUT these tools may vary with the target.
You asked more so how about Linux to some Android phone? https://www.google.com/search?q=install+linux+on+android+phone seems to get into it.
-> But this is where we get into deep waters. If you want to make your own OS and apps, we won't be seeing you for decades. I suggest you take a bog standard Linux that was used in the linux on android before and step 1, learn to compile that version and step 2, alter it to do what you want.
The thing is, here in the NSA (country formarly known as the USA) the government has taps all over. While Yahoo got slammed over writing search code for this country, these taps were dictated before.
Please read https://en.wikipedia.org/wiki/Lavabit as it revealed those taps years ago. Levison was under a gag order and eventually closed up shop.
Given the taps all over on internet, email, phone, SMS and more, just how do you propose to solve this while communicating with others?
-> If you want to open a discussion about secure comms, let's do that! For me the old ways can be the best ways.
So let's say you and I want to securely message each other. Let's make an app that uses a codebook. Since there are so many, we have to first agree on a codebook then our apps fetch those books and then... You get the idea.
Of course it is possible. The question is why?)