You must also take into account the valence bond of the different elements !
For example : a you can write a simple formule like CH2 but this can never exist as an molecule on it's own. A carbon atom has 4 "hooks" so CH4 can exist as a molecule. CH2 cannot. When the number of atoms increase so will their combinations most of wich cannot be a real molecule.

Don't think this idea'll fly. As mentioned, the number of permutations will be staggering, especially if all elements (below or equal to the molecular/formula mass) can be valid. Even organic molecules could be metallo-organic, so it doesn't help v. much unless you specify the constituent elements of the compound.

Also, I assume you're referring to Mr/Fr rather than mass in grams??!

The fact that C can form triple, double bonds as well as multiple aromatic rings (heterogeneous/homogeneous) as well as alicylic rings, will just confuse matters. I don't think you'll find an algorithm to weed out the possible and impossible answers.