That function does not change the size of the buffer pointed to by lpBuf, actually it can't because lpBuf is passed by value.
The code of the function implies that dwBufLen must be <= the size of the buffer pointed to by dwBufLen.
You should consult the creator of this function or the creater of the code that calls this function to find out for sure if the meaning and use of dwBufLen is the length of the buffer pointed to by lpBuf; although by normal convention it would be.
That function does not return a buffer, it accepts a buffer (well 2 buffers really, lpBuf and lpKey) and a parameter (again 2 really for the size of each of the buffers) that by convension is the length of the buffer.
Convention is that the parameter passed in dwBufLen is the length of the buffer pointed to by lpBuf. However that is entirely dependent on the calling code, there is nothing to prevent the calling code (which you don't show) passing a pointer to a buffer that is longer than dwBufLen. However whatever the actual length of the buffer that function on works on the first dwBufLen BYTEs of it.
If that function is doing any compressing then is is compressing the data in lpKey into lpBuf. However the function does not change the size of either of the buffers lpKey and lpBuf.