Using Named Pipes with Asynchronous I/O Redirection to WinAPI

[Dennis M.]

The title in and of itself is fairly descriptive, however, I am looking for some help. I am writing a program which reads the output of a command-line based program and puts that output to a GUI. Due to my need of asynchronous i/o, I have had suggestions to use named pipes. Before, I could read from my anonymous pipes, but I had the issue of reading continuously (which is why I'm switching to named; waitforallobjects(); etc). So I have the named pipe properly setup and I am creating the process (with CreateProcess()) in the same fashion I was with an anonymous pipe, however, I cannot extract the output. Any ideas?

Regards,
Dennis M.

[wisaacs]

Named pipes will work. Nothing you said is wrong, so you will have to post code. The assumption that pipes are required for asynchrony is wrong though. Pipes have no relationship to asynchrony. You could do everything in one process without IPC (interprocess communication), by launching a thread from your WPF or Forms application. You can enable the console system on a GUI project, and you automatically get a console window which responds to standard io commands.

[Dennis M.]

Thanks for your response! I haven't looked so much into trying to multi-thread this program I am working on because I am taking the i/o directly from another program that I did not write but which outputs to a console window. Here is the relevant code

#define INSTANCES 3
#define CONNECT_STATE 0
#define READ_STATE 1
#define WRITE_STATE 2
#define BUFSIZE 0x1000 // 4096 bytes

void Arc_Redirect::createProcesses()
{
	TCHAR programName[]=TEXT("program.exe");
	PROCESS_INFORMATION pi; 
	STARTUPINFO si;
	BOOL bSuccess = FALSE; 

	// Let's use named pipes <3
	for(int i=0;i<INSTANCES;i++)
	{
		hEvents[i] = ::CreateEvent(NULL, TRUE, TRUE, NULL);

		if(hEvents[i] == NULL)
			throw "Could not init ArcEmu!";

		outStd[i].o1.hEvent = hEvents[i];

		outStd[i].hPipeInst = ::CreateNamedPipe(
			TEXT("\\\\.\\pipe\\arcworld"), PIPE_ACCESS_DUPLEX | FILE_FLAG_OVERLAPPED, PIPE_TYPE_MESSAGE | PIPE_READMODE_MESSAGE | PIPE_WAIT,
			INSTANCES, BUFSIZE*sizeof(TCHAR), BUFSIZE*sizeof(TCHAR), PIPE_TIMEOUT, NULL);

		if(outStd[i].hPipeInst == INVALID_HANDLE_VALUE)
			throw "Could not init ArcEmu!";

		outStd[i].pendingIO = getState(outStd[i].hPipeInst,&outStd[i].o1);

		outStd[i].dwState = outStd[i].pendingIO ?
			CONNECT_STATE : READ_STATE;
	}

	// Set stuff up
	ZeroMemory( &pi, sizeof(PROCESS_INFORMATION));
	ZeroMemory( &si, sizeof(STARTUPINFO));
	si.cb = sizeof(STARTUPINFO); 
	si.hStdError = outStd[READ_STATE].hPipeInst;
	si.hStdOutput = outStd[READ_STATE].hPipeInst;
	si.hStdInput = outStd[WRITE_STATE].hPipeInst;
	si.dwFlags |= STARTF_USESHOWWINDOW|STARTF_USESTDHANDLES|FILE_FLAG_OVERLAPPED;

	CreateProcess(programName,NULL,NULL,NULL,FALSE,0,NULL,NULL,&si,&pi);
}

BOOL Arc_Redirect::getState(HANDLE hPipe, LPOVERLAPPED lpo)
{
	BOOL connected, pendingIO = FALSE;

	// Overlap connection for this pipe
	connected = ::ConnectNamedPipe(hPipe,lpo);

	if(connected)
		throw "ConnectNamedPipe(); failed!";

	switch(::GetLastError())
	{
		case ERROR_IO_PENDING:
			pendingIO = TRUE;
		break;
		case ERROR_PIPE_CONNECTED:
			if(::SetEvent(lpo->hEvent))
				break;
		default:
			throw "ConnectNamedPipe(); failed!";
		break;
	}
	return pendingIO;
}

void Arc_Redirect::readPipe()
{
	char chBuf[BUFSIZE];
	::ReadFile(outStd[READ_STATE].hPipeInst,chBuf,BUFSIZE,&outStd[READ_STATE].dwRead,&outStd[READ_STATE].o1);
	chBuf[outStd[READ_STATE].dwRead] = '\0';
	::SetDlgItemText(global,IDO_WORLDOUT,chBuf);
}

outStd is a struct defined within the class. Here is that struct:

typedef struct
	{
		HANDLE hPipeInst;
		OVERLAPPED o1;
		TCHAR chReq[BUFSIZE];
		TCHAR chReply[BUFSIZE];
		DWORD dwRead;
		DWORD dwWritten;
		DWORD dwState;
		DWORD cbRet;
		BOOL pendingIO;
	} PIPE_HANDLES, *LPSTDPIPE;

Any ideas?

Thanks!
Dennis M.

[wisaacs]

Here is the real answer, and it was very tough to discover.

Named Pipes normally fail for redirection because the default handles must be modified before they are inherited by the second process.

Your server-side handles need to be non-inheritable, even though you must create the pipe using an inherit-handle attribute. The solution is, you first create the pipe, then duplicate both handles with the attributes fixed, then close the original handle.

// you need this for the client to inherit the handles
	SECURITY_ATTRIBUTES sa;
	// Set up the security attributes struct.
	sa.nLength= sizeof(SECURITY_ATTRIBUTES);
	sa.bInheritHandle = TRUE; // this is the critical bit
	sa.lpSecurityDescriptor = NULL;
	//
	// Create the child output named pipe.
	//
	//  anon. pipe would be:
	//  if (!CreatePipe(&hOutputReadTmp,&hOutputWrite,&sa,0)) DisplayError("CreatePipe");
	//
//
// this creates a inheritable, one-way handle for the server to read
//
	if((hOutputReadTmp= CreateNamedPipe(
		"\\\\.\\pipe\\outstreamPipe",
		PIPE_ACCESS_INBOUND,
		PIPE_TYPE_MESSAGE|PIPE_READMODE_MESSAGE|PIPE_WAIT,
		1,
		4096,
		4096,
		0,
		&sa
		))==INVALID_HANDLE_VALUE) DisplayError("CreateNamedPipe");
//
// however, the client can not use an inbound server handle.  Client needs a write-only,
// outgoing handle.
// So, you create another handle to the same named pipe, only write-only.
// Again, must be created with the inheritable attribute, and the options are important.
//
// use CreateFile to open a new handle to the existing pipe...
//
	hOutputWrite = CreateFile(
		"\\\\.\\pipe\\outstreamPipe",
		FILE_WRITE_DATA|SYNCHRONIZE,
		0,
		&sa,
		OPEN_EXISTING, // very important flag!
		FILE_ATTRIBUTE_NORMAL,
		0 // no template file for OPEN_EXISTING
		);
	// we're going to give the same pipe for stderr, so duplicate for that:
	// Create a duplicate of the output write handle for the std error
	// write handle. This is necessary in case the child application
	// closes one of its std output handles.
	if (!DuplicateHandle(GetCurrentProcess(),hOutputWrite,
		GetCurrentProcess(),&hErrorWrite,0,
		TRUE,DUPLICATE_SAME_ACCESS))
		DisplayError("DuplicateHandle");

// All is well?  not quite.  Our main server-side handle was created shareable.
// That means the client will receive it, and we have a problem because 
// pipe termination conditions rely on knowing when the last handle closes.
// So, the only answer is to create another one, just for the server...
	// Create new output read handle and the input write handles. Set
	// the Inheritance property to FALSE. Otherwise, the child inherits the
	// properties and, as a result, non-closeable handles to the pipes
	// are created.
	if (!DuplicateHandle(GetCurrentProcess(),hOutputReadTmp,
		GetCurrentProcess(),
		&hOutputRead, // Address of new handle.
		0,FALSE, // Make it uninheritable.
		DUPLICATE_SAME_ACCESS))
		DisplayError("DuplicateHandle");
// now we kill the original, inheritable server-side handle.  That will keep the
// pipe open, but keep the client program from getting it.  Child-proofing.
	// Close inheritable copies of the handles you do not want to be inherited.
	if (!CloseHandle(hOutputReadTmp)) DisplayError("CloseHandle");

.. I'll skip the part where you do the same thing for client's standard-input, and get to the launcher...

PROCESS_INFORMATION pi;
	STARTUPINFO si;

	// Set up the start up info struct.
	ZeroMemory(&si,sizeof(STARTUPINFO));
	si.cb = sizeof(STARTUPINFO);
	si.dwFlags = STARTF_USESTDHANDLES;
	si.hStdOutput = hOutputWrite;
	si.hStdInput  = hInputRead; // I didn't give you this one.. exercise!
	si.hStdError  = hOutputWrite;
	// Use this if you want to hide the child:
	//     si.wShowWindow = SW_HIDE;
	// Note that dwFlags must include STARTF_USESHOWWINDOW if you want to
	// use the wShowWindow flags.


	// Launch the process that you want to redirect (in this case,
	// an app I wrote that just echoes everything you type with
	// "<what you wrote>?  After you, my dear Alphonse!"
	if (!CreateProcess(NULL
		,"D:\\src\\afterYouMyDearAlphonse\\Debug\\afterYouMyDearAlphonse.exe"
		,NULL,NULL,TRUE,
		CREATE_NEW_CONSOLE,NULL,NULL,&si,&pi))
		DisplayError("CreateProcess");

	// Close the useless handles.
	if (!CloseHandle(pi.hThread)) DisplayError("CloseHandle");
	if (!CloseHandle(pi.hProcess)) DisplayError("CloseHandle"); // this could be used for waits

Immediately after launching the child process, close the handles you are holding which only the child is supposed to use.

// Close pipe handles (do not continue to modify the parent).
	// You need to make sure that no handles to the write end of the
	// output pipe are maintained in this process or else the pipe will
	// not close when the child process exits and the ReadFile will hang.
	if (!CloseHandle(hOutputWrite)) DisplayError("CloseHandle");
	if (!CloseHandle(hInputRead )) DisplayError("CloseHandle");
	if (!CloseHandle(hErrorWrite)) DisplayError("CloseHandle");

.. and now you can read from the non-inherited handles

CHAR lpBuffer[256];
	DWORD nBytesRead;
	DWORD nCharsWritten;

	ConnectNamedPipe(hOutputRead,0);

	while(TRUE)
		{
		if (!ReadFile(hOutputRead,lpBuffer,sizeof(lpBuffer),
			&nBytesRead,NULL) || !nBytesRead)
			{
			if (GetLastError() == ERROR_BROKEN_PIPE)
				break; // pipe done - normal exit path.
			else
				DisplayError("ReadFile"); // Something bad happened.
			}

		// Display the character read on the screen.
		if (!WriteConsole(GetStdHandle(STD_OUTPUT_HANDLE),lpBuffer,
			nBytesRead,&nCharsWritten,NULL))
			DisplayError("WriteConsole");
		}
	}

Of course, you need to send your keyboard output to the client's standard-input, which requires a worker thread to sample and send on the other pipe.

But, that is another thread.

[wisaacs]

correction in STARTUPINFO initialization block:

// Set up the start up info struct.
	ZeroMemory(&si,sizeof(STARTUPINFO));
	si.cb = sizeof(STARTUPINFO);
	si.dwFlags = STARTF_USESTDHANDLES;
	si.hStdOutput = hOutputWrite;
	si.hStdInput  = hInputRead; // I didn't give you this one.. exercise!
	// THIS IS THE CORRECTION:
	si.hStdError  = hErrorWrite; // use the duplicate handle created for stderr