Posts by JamesCherrill

Hi musthafa.
The code I posted gets the coords from the JLabel mouseevent and and passes them as two ints to the server. That part works OK, even when the JLabel is being scrolled in a JScrollPane. You will see that there's no need to convert the origin as your code does on lines 3,4.

can you help me in your way how do send keyboard and mouse event from client side(screenwatcher) to server(screen sender..)

I'm currently working on a simple addon to send mouse clicks and keyboard presses to the server. I'll post what I've got so far - but it's not complete and not properly tested. In particular I think I may have a thread locking problem at the server between using Robot for screen capture and for mouse/key events on different threads, but I really don't know yet.
So, at your own risk, this is what I've got so far...
At the client I add the following mouse and keyboard listeners to the JLabel that displays the screen ("out" is tha buffered data output stream)

@Override
	public void mousePressed(MouseEvent e) {
		// System.out.println("Mouse button " + e.getButton() + " pressed at " +
		// e.getX() + "," + e.getY());
		try {
			out.writeInt(MOUSE_PRESSED);
			out.writeInt(e.getButton());
			out.writeInt(e.getX());
			out.writeInt(e.getY());
			out.flush();
		} catch (IOException e1) {
			e1.printStackTrace();
		}
	}

	@Override
	public void mouseReleased(MouseEvent e) {
		// System.out.println("Mouse button " + e.getButton() + " released at " +
		// e.getX() + "," + e.getY());
		try {
			out.writeInt(MOUSE_RELEASED);
			out.writeInt(e.getButton());
			out.writeInt(e.getX());
			out.writeInt(e.getY());
			out.flush();
		} catch (IOException e1) {
			e1.printStackTrace();
		}
	}

	@Override
	public void keyPressed(KeyEvent e) {
		// System.out.println("Key " + e.getKeyCode() + " pressed");
		try {
			out.writeInt(MOUSE_LAST + 1);
			// my code for key released (shouldn't clash with mouse codes)
			out.writeInt(e.getKeyCode());
			out.flush();
		} catch (IOException e1) {
			e1.printStackTrace();
		}
	}

	@Override
	public void keyReleased(KeyEvent e) { …

Yes, the first frame is always going to be a big hit, and a colourful background is going to kill the GIF/PNG type compressions that you need for text etc. A 1280*1024 24 bit colour image is 3840k of raw data, so 3380k isn't surprising. For certain, replacing the background picture with a solid colour will give a big improvement. You're right - that IS why commercial products like to grey out the background. Maybe, just for the very first screen, it would be better to revert to the PNG version, then switch to my differential RLE thereafter???
musthafa's code (which I don't really understand) uses JPEG, which will be good for the background picture, but horrible for text - I think JPEG is not appropriate.

The question seems ambiguous to me.
Any Java interface can contain many methods.
What this more probably refers to is illustrated in this example:
You have an interface Vehicle that declares a void drive(int speed) method.
class Car implements Vehicle {
void drive.... etc
so does
class Speedboat implements Vehicle {
void drive(...
Now you have one interface, two versions of the method. You can simply call the method, and the appropriate implementatoin is used:
Vehicle v = (something...)
v.drive(30); // calls Car's drive or Speedboat's drive depending on what kind of Vehicle this is.

Have a question... close the thread?

I think that depends on what you find. If you think it's as good as we're going to get then, yes, we're all done with ths topic. If not, let's keep going until we have a good answer :-)

Aaaarggggg! Sorry Guys, the previous post has the WRONG VIEWER code.
Here's the correct one.

OK Guys, here you are - actual source code.
As before, you need to establish a socket-socket connection and pass the sockets to the server & client classes.
For fastest possibel refresh, set the targetRefreshInterval to 0.
With nothing much happenening on the screen the updates are now as small as 1 - 2KB. I can't believe we can do much better than that!
Have fun, let me know how you get on.
James

Yes, this is based on your idea of pixel change, but instead of transparency I just send a code of how many pixels to leave unchanged. The (current) comments from the code may help...

// Output stream format is sequence of 1 integer records describing the
// data array in natural order (element 0 ... data.length-1)
//
// Unchanged record (sequence of >=1 unchanged pixels):
// bit 0 is 1,
// bits 8-31 are number of consecutive unchanged pixels.
//
// Changed record (sequence of >=1 identical changed pixels):
// bit 0 is 0,
// bits 1-7 are number of consecutive pixels the same,
// bits 8-31 are the 3 byte RGB values for these pixels.

... I don't undestand why "your reputation hangs on this"?
Thx.

I was just that if this didn't work I'd look like a fool - nothing to do with DaniWeb!
This version takes the current array of ints and the previous version, and compresses into a single stream. Consecutive runs of unchanged pixels are coded as a single int (ie number of pixels unchanged), changed pixels are compressed by ordinary RLE. At the client the previous int array buffer is kept, and only the changed values are overidden. I then re-create the image from that buffer each time. No PNG, no transparency, so its simple & fast.
I've just got it working, so I'll clean it up a bit and post the source tomorrow.
ps You should be able to use those jars over the internet, just give the viewer the IP address of the server when it asks. The server listens on port 1234, so you may need to open that thru the firewall and/or router.

OK, my repution hangs on this one...
Server & viewer jars as before but with my latest non-tiled RLE compressed algorithm. This also compresses runs of consecuitive pixels that have not changed into a single int. Update volumes are down to 6kB when nothing much is happening, refresh rates are still up to 5 per sec without the delays.
Let me know, OK?

If you are using my RLE compressor, that was written assuming that the Alpha value was not being used - so it uses that byte to hold repeat counts, and forces it back to Alpha = 255 (100%) when decompressing. Obviously that's not going to work if you want to send transparent pixels and keep their transparency!
I have an idea for incorporating the changed/unchanged info in my RLE compressor that may just work very well. I'll try to do some work on it, but I'm a bit busy at the moment. I'll get back to you when I get a chance.

maybe need to do a setData to re-write the raster to the image?

Is line 11 the right way round (don't you want to change pixels2)?

I found it about twice as fast, plus there's zero overhead converting back to image. Good Luck!

createImage is defined as returning Image (abstract) but actually returns a BufferedImage, so you can just cast it.
However, rather than extracting pixels, prcessing them and converting back to Image, why not work directly with the Imaage's databuffer array and save both conversions:

image = robot.createScreenCapture(new Rectangle(width, height));
Raster ras = ((BufferedImage) image).getData();
DataBufferInt db = (DataBufferInt) ras.getDataBuffer();
int[] data = db.getData();

now whatever you change in data will change the image directly.

I used the following standard solution:

ims = new MemoryImageSource(width,height, pixels, 0, width);

(where the width/height vars define the image size and pixels is a simple int array of PixelGrabber-type data

then:

Image im = Toolkit.getDefaultToolkit().createImage(ims);

It seemed to me that this was fast enough to be be negligeable compared to moving the pixel aray over the network etc.

I would try the second idea. Don't use my RLE because that uses the alpha byte, and we need the alpha byte intact. Use PNG. And no, there's no reason to split the image with that. I think it's got great potential.
I'm offline now till tomorrow.

If I understand correctly - you dont need to do the stuff with a byte array to use png - the following should be simpler and faster:

OutputStream out = clientSocket.getOutputStream();
ImageIO.write(image, "png", out);

and, at the client end...

Image remoteImage = ImageIO.read(socket.getInputStream());

Hi Clawsey. Interesting ideas!
1. Shouldn't be too hard to try this - there's a loop that looks for changes between the prevoius and latest pixels for each cell/tile, so that's a place where the lowest& highest x&y can be recorded to define the smallest rectangle containing all the changes. On the other hand, with 256 tiles each one is pretty small, and there may not be much to gain. I'm still wondering what speed internet link you are using, and why the data rate seems so low (is there a high latency? - what's the ping?).
2. Each pixel already has an alpha value in its first byte, set that to 0 and the pixel is totally transparent. So you could do as you say and set all the unchanged pixels to alpha=0. You can then set the RGB values to some constant value (because they don't matter for totally transparent!) and compress the whole image as PNG which is ultra-efficient at compressing runs of identical pixels. I think the is potentially a really great idea, and should be pretty easy/small to code.
I can give you the code for sending and receiving an Image with PNG compression if you like - it's really small.

We've all done it sometime :-) Mark this is solved?

Line 3 you confuse q and i.

On internet you can see the labels delay... and if you write something from server screen (about 3,4 modified cells right?) you get about 2 seconds delay.

This is puzzling. I've added some more statistics to the server, and when nothing is happening much on the screen the updates are under 100kB each. That shouldn't take 2 seconds unless you're on a modem connection!
I've attached the version with these stats, no delays, and with the tiles back to 256 tiles/screen (80*64 pixels) so we can see
what that tells us on your internet connection.

Thanks... work very well on LAN for cell number from the first example (you modified it later no?)

Earlier I was dividig the screen into 16X16 tiles (better word than "cells"). Later I found a better trade-off dividing into 8x8 tiles (ie about 160x128 pixels each). The tile size is definitely an interesting thing to play with, as more/smaller tiles increase the CPU usage on the server, but reduce the amount of data to send. In my case I'm more worried about server CPU load.

I get this output from the sender:
Sending
Compressed (9%)
...
Compressed (17%)
Updated in 14813 mSec

Looks like an unbuffered output stream to me! I write one word at a time, so unbuffered writes are a disaster.
I'll get rid of the compression stats output now, they have served their purpose.

i taken it into my account..

i will make it for my app suit..

musthafa: You have my permission to use my code in your application, but you must place a comment
// uses code provided by James Cherrill 2010
in your program near where my code is used.
Cheers
James

did you change somthing in this last demo? This demo is indeed fast.

Only change is that I took out the sleep(...) waits that I used to throttle the previous version back to one update every 2 seconds.

Here's a jar without any frame rate limiting - this will send updates as fast as it can. See how that goes?

it working slower than i have even at LAN
(2 or 3 seconds...)..
we worried at INTERNET performance...

It's set to send an incremental refresh every 2 secs (compromise between responsiveness and server CPU loading). That's configurable. At that speed it should be using <10% CPU, and giving a negligable LAN network load.
I understand the internet issue. A full screen image, with lossless compression, is still going to be near a megabyte, and on a 2 megabit ADSL line that's an unavoidable 4 seconds!
All you can do is to minimise the number of bytes transmitted, which is what I've tried to do, without corrupting the screen image with lossy encoding like JPEG.

ps: Just tried without the 2 second refresh rate limit (ie target rate 0), gives me 5 frames/second (200 mSec per refresh), LAN still nowhere near saturated.

Hi guys.
Don't know why yu get these delays - I'm seeing 200mSec to scan the whole screen and then just the transmission delays for a few changed cells (64*64*4 bytes each, less 30% for RLE compression) = 11k/changed cell.
Across a WiFi link with a 2-second refresh rate it's almost like watching the source screen, and server CPU is < 10%.
I've stuffed the code together with just enough context/infrastructure to run (I'm doing this within quite a large context, all of which is irrelevant here).
The zip contains a server jar and a viewer jar.
The server has no UI, and exits when the client disconnects. If you can't connect in the first place you'll have to crash it from taskmanger or whatever.
The client attempts to re-connect to a server at whatever address it previously found it, and prompts for a host name / IP address if it can't connect (eg first time run).
Let's see how that works for you!

i seen some solution regarding make 16 x 16...
still processing of every rectangle also take some seconds....

Yes, I found the processing at the server was the limiting factor, and small cells made it worse. I did a lot of optimisation on the server code - replacing PixelGrabber with direct access to the images' raster#'s databuffer, and moving all the code I could out of the loops. I got the best trade-off between processing time and update transmission time with cells around 64x64.

//first run
...
//second run
...

Not sure what you mean here - its designed to be run just once - it sits waiting for updates and displays them as and when they arrive. If you close the window and start again, you should use a new socket connection.

Haven't seen that myself. That's where the client reads the RLE compressed array - first reads the length then it allocates an array of that size. Looks like its getting the wrong int in the readInt for the length, but I don't know why. That number looks like one of the RGB data ints.
It starts by sending the cell sizes etc - do these get across OK? (if so, you will see a blank display window of the right size (probably with scroll bars).
Is it possible that anything else is interfering with the datastream?

OK Clawsy - have a go with these.
The server class assumes you already have the socket connection (passed into constructor)
The client (viewer) class has a call to another of my routines to set up the socket - you'll need to replace this.

ps: server currently sends screen (updates) then waits 3 secs and repeats until client closes. You will need to do something smarter - eg run every 1 sec? The server does the activity logging to System.out

OK, I'll put together the appropriate bits and a few notes and post later (may be tomorrow). Transmission should be fast - only send changed blocks, just send raw pixels as ints, and used RLE encoding that gives a further reduction in sent data of typically 30%+ depending on the image (best with plain backgrounds!).

In case anyone's still interested in this, I tried a few things and got the best result by dividing the screen into 16x16 equal rectanges and only sending the ones that changed. After some optimisation I got down to 1/5 second (200 mSec) to scan the whole screen and identify the changed tiles. I send that data as an array of raw integers (RGB) via a buffered data stream - giving a round-trip elapsed time of just over 1 sec (LAN) for a complete screen (first time only!), reducing linearly to zero with the number of changed rectangles. I used a grid ot JLabels to display the screen at the client, thius avoiding having to merge the changed rectangles myself. I can share the critical code if this is still a live issue for anyone.
Peace.
J

... Maybe just split the screen in 4 and scan every one of them, then send a maximum number of 4 rectangles.

This takes us back to the previous idea of dividing into sub-rectangles (I suggested 8x8 pixels) and only sending the changed ones. Obviously you can play with the size vs number of sub-images - maybe 80 blocks 128*128 would be better. IMHO this is still the best way to reduce the number of kbytes to transfer (I would still use ImageIO/png to send the sub-images).

Yeah, its a lan connection - server 100 ethernet to router, client WiFi g to router. Tehre's nothing special in the code... I started by sending the image via an Object Stream, but that was nearly 3 secs. The "final" version uses ImageIO.write in png format.
I think the connection speed is the limiting factor, so obviously a 1 meg ADSL link would slow it right down!

After our discussions I realised that I could make good use of a similar thing myself, so I did a quick implementation and have the following performance info for you:
The server is a low-end desktop, 1280x1024 screen.
The client is an old laptop, WiFi g connection.
The process being bechmarked is as follows:
Client opens new socket connection to server
Client sends request to server.
Server uses Robot to get full screen image
Server sends image to client
Client closes connection.
Client displays image in scrolling window

End to end timing for that whole process is under a second, but I haven't tried to break down the individual parts.
<1 sec is more than good enough for me, and compares well with commercial remote desktop software I have used. What performance goal do you have?

Where's the slowdown - is it in the getSubImage or in the sending? - because if it's in the sending I can't see how you would speed that up (I assume you're using one connection for all the subImages, not creating a new connection for each)
Since you have the subimage already as a string of consecutive ints (in the grabber array) maybe it would be faster to send that and reconstruct the image at the receiver?

Multiple rectanges: I hate the idea of trying to find a variable number of rectanges of variable size that minimise the total area that conatins all modified pixels. Sounds like a programming nightmare, and the overheads will probably exceed he gains.
But: how about dividing the image into fixed 8x8 pixel cells and finding the cells that have changed? That's like MPEG, and really fast & easy.

boolean[][] cellChanged = new boolean[w >> 3][h >> 3];

int index = 0;
for (int x = 0; x < w; x++) {
	for (int y = 0; y < h; y++) {
		if (array1[index] != array2[index])
			cellChanged[x >> 3][y >> 3] = true;
		index++;
	}
}
	
for (int y = 0; y < (h >> 3); y++) {
	for (int x = 0; x < (w >> 3); x++) {
		System.out.print(cellChanged[x][y] ? "1" : "0");
	}
	System.out.println("");
}

Now you've also got an easy starting place for transmitting only the cells that have changed, and each cell is small enough that you can just transmit it all.

I don't think multiple threads will help much unless you have a quad processor! You already have GUI and socket IO running (presumably) on different threads. I'm still surprised by your lomg timings - my 20 mSec was on a 1.66 GHz core 2 laptop.

Also, in your benchmark code you grab both sets of pixels - in the real case you can just grab the latest image and pass the previous one forwards.

You'll notice I replaced the i+j*w repeated calc with a simple index counter. That reduced my benchmark from 68 ms to 20. Suggest you try that.

Just for fun I created two int arrays 1280*1024 and compared them in a simple nested loop.
The followin code executes in 20 milliSecs on my old laptop

final int w=1280, h=1024;
int[] array1 = new int[w*h];
int[] array2 = new int[w*h];
long start = new Date().getTime();
int mismatches = 0;
int index = 0;
for (int x = 0; x< w; x++) {
	for (int y = 0; y< h; y++) {
		if (array1[index] != array2[index]) mismatches++;
		index ++;
	}
}
System.out.println(new Date().getTime() - start);

Just get a 1024*1280 image and try it ! It's only a million integer compares - on a multi-Gig processor.
It's such a common mistake that it has its own name: "premature optimisation".
Confirm that it really is a problem, then let's discuss optimisations.

Dunno what you mean by "infinite values", but maybe you should get things working one at a time, starting with easy data.
What I would do is to create two small bitmap images (no compression artefacts) say 16 * 8 pixels with just a couple of known pixels different, and use these to test my compare routine.
When that's working 100% I'd move on to bigger images until I was at full size. From that I'd get a relationship between image size and execution time, and have a test bed for any perfomance tuning.
I wouldn't attempt to do any of the above in the context of the full working program.

line 8:
the index into the arrays is i + j* w
(the arrays ate in row order)

Sorry, but you need to read again. You have been given the solution.

mseck: nobody wants to insult anyone, but it is very frustrating when people don't listen. javaAddict told you how to fix that problem earlier, and reminded you in his last post.
I know its frustrating when you get a compile error you don't understand, but don't let that get in the way of reading and thinking.

Hi javaAddict: I hadn't realised that you were already dealing with mseck's problem. (And, once again, I apologise for my "senior moment" with the + , even worse now I've read the previous thread!).
I gonna bow out now, before I make a bigger fool of myself.

What's the exact code on the line where you get the error?