How would I execute a C/C++ program without using a windows library? (Like ShellExecute)

In assembly? You can (maybe) but is it worth the effort you'll need to write the loader?

The OS exists to make complicated things simple. Just use the OS functions.

Yes, in asm.

That's the problem, I'm making an OS, and I don't want it to depend on any other OSes. (except for DOS is fine)

I tried using INT 40h, but it doesn't seem to work. The code compiles and runs, but nothing happens on the interrupt.

Ah, well, then you've got yourself in a ball of mud.

One of the primary functions of the OS is to load and execute a program. How this happens is typically and entirely dependent on the OS.

For example, Win32 cannot execute a Mac program. As far as old DOS and Win16 programs, Win32 was specifically engineered to support legacy applications, but the next generation will scrub the old DOS subsystem anyway...

If you intend to execute PE32 (Win32) programs, then you'll have to do some serious reading over at MSDN and around the net.

Otherwise, you'll have to roll your own exe format, or borrow one from another OS, and write your own loader.

Good luck.

Maybe consider writing a small boot loader and use BIOS calls instead. Most of the functionality you need is provided by any of the PC BIOS's. That way you are completely independent of any operating system including DOS. As your project develops, then inevitably you can get rid of any BIOS dependence too like Linux

Okay, I will try to go about it a little differently, or do as you said.

Oh, I didn't see your post, Tight. I will look into that as well.