Implementing a new protocol

[purijatin]

Hello
For example if i want to implement my own protocol, i can proceed by implementing the extra features on UDP on the application it self and thus achieving my task. But what if i want to implement it on the OS level just like TCP or any other protocol, i mean for example in Java we have a in built library which can provide TCP and UDP. Now how do these libraries provide the facility, do they implement it them selves , if yes then how do they connect to the network layer. Or else do they forward it to the Operating System which implements the TCP, for them. If not then who implements it and send the data to the network layer
Thanks for the help