... Maybe just split the screen in 4 and scan every one of them, then send a maximum number of 4 rectangles.
This takes us back to the previous idea of dividing into sub-rectangles (I suggested 8x8 pixels) and only sending the changed ones. Obviously you can play with the size vs number of sub-images - maybe 80 blocks 128*128 would be better. IMHO this is still the best way to reduce the number of kbytes to transfer (I would still use ImageIO/png to send the sub-images).
Hehe, a lot of strugle with firewall I had, depends on your router, forwanding.. can't say nothing now.
About the idea with spliting... you don't knwo how I get the rectangle, that's why you don't undestand my idea. I get the minimum coords and maximum coords of the rectangle. Thinks I woudl have screnn splited int 2. I get the menu rectangle at the left and another rectangle with the clock at right = 2 small images. If I don't split it I woudl have a much more bigger rectagle with the minimum x,y at the menu left-top and the maximum x,y at the clock right bottom. Hard to explain...
ok then leave it...
again it not so easy to find 4 rect by make it min,max cord by notifying it...
definitely we should goto win api for this...but again it will make it OS dependancy...
have u ever looked mikogo...
any how for firewall issue and image definitely Win api COMES TO PLAY..
so whhhhaaaaaaaattttttttt with JAVA still Irritating....:@
In case anyone's still interested in this, I tried a few things and got the best result by dividing the screen into 16x16 equal rectanges and only sending the ones that changed. After some optimisation I got down to 1/5 second (200 mSec) to scan the whole screen and identify the changed tiles. I send that data as an array of raw integers (RGB) via a buffered data stream - giving a round-trip elapsed time of just over 1 sec (LAN) for a complete screen (first time only!), reducing linearly to zero with the number of changed rectangles. I used a grid ot JLabels to display the screen at the client, thius avoiding having to merge the changed rectangles myself. I can share the critical code if this is still a live issue for anyone.
Peace.
J
Of course I am interested :). I guess you didn't test it on Internet no? However I am interested in the code. Very :). I can share some code too if you want... it would be nice to test this on Internet. Maybe I can optimize it further with the transmision taking some parts of your code... and then share it back to you, of course. Thanks.
P.S.: If I wouldn't be interested I would say it.
OK, I'll put together the appropriate bits and a few notes and post later (may be tomorrow). Transmission should be fast - only send changed blocks, just send raw pixels as ints, and used RLE encoding that gives a further reduction in sent data of typically 30%+ depending on the image (best with plain backgrounds!).
OK Clawsy - have a go with these.
The server class assumes you already have the socket connection (passed into constructor)
The client (viewer) class has a call to another of my routines to set up the socket - you'll need to replace this.
ps: server currently sends screen (updates) then waits 3 secs and repeats until client closes. You will need to do something smarter - eg run every 1 sec? The server does the activity logging to System.out
Thanks. I will spend some time to use some of your code :)... because I had a totally different way to send the image, and I want to see if yours will be faster. (maybe it will). I just made array of bytes from the image and then send it and reconstruct it from array to image at the client. Your way seems interesting. So after I decide myself about your code, I will post back... maybe tomorow.. we'll see.
Thanks again.
James, I have an Java Heap Space (out of memory) Exception at this line of the viewer:
int[] temp = new int[length];I printed the length: 2113951979
:). Did you met this problem or I did not embed your code as expected?
Haven't seen that myself. That's where the client reads the RLE compressed array - first reads the length then it allocates an array of that size. Looks like its getting the wrong int in the readInt for the length, but I don't know why. That number looks like one of the RGB data ints.
It starts by sending the cell sizes etc - do these get across OK? (if so, you will see a blank display window of the right size (probably with scroll bars).
Is it possible that anything else is interfering with the datastream?
Don't know... mabe its me.. here is an output:
cells:160x128//first time
//first run
i=0 j=0
2035
//second run
i=0 j=344
2113951979
I should miss something... because if it worked for you... should work for me too. I will check...
i seen some solution regarding make 16 x 16...
still processing of every rectangle also take some seconds...
it was delayed further..
any how i also part of your work..
i will try with your ideas..and come back...
//first run
...
//second run
...
Not sure what you mean here - its designed to be run just once - it sits waiting for updates and displays them as and when they arrive. If you close the window and start again, you should use a new socket connection.
i seen some solution regarding make 16 x 16...
still processing of every rectangle also take some seconds....
Yes, I found the processing at the server was the limiting factor, and small cells made it worse. I did a lot of optimisation on the server code - replacing PixelGrabber with direct access to the images' raster#'s databuffer, and moving all the code I could out of the loops. I got the best trade-off between processing time and update transmission time with cells around 64x64.
I will try to make it work... sorry for delay, I have a lot to study. I will try to integrate your code in my app... then post back.
I made it work but its so slow, as I can see every cell painted one by one about 0.5 second every one of them. Maybe it's my bad? Wrong data flushing? If it worked fast for you the problem is mine... 'Updated in 19453 mSec'
Here is my code customized:
Sender (slightly modifued, just to use my DataInsputStream and my socket)
class ScreenSender extends Thread {
// helper class to send and update screen images in real time
Robot robot = null;
DataOutputStream out = null;
int width = 0, height = 0;
int cellWidth = 0, cellHeight = 0;
int cellsAcross = 0, cellsDown = 0;
int[] pixels = new int[0];
int[][][] prevCells;
BufferedImage image = null;
Socket clientSocket;
ScreenSender(chatServer cs) {
clientSocket = cs.clientSocket;/***** modified/added here ********/
out = cs.os_img;/***** modified/added here ********/
try {
robot = new Robot();
Dimension screenSize = Toolkit.getDefaultToolkit().getScreenSize();
width = screenSize.width;
height = screenSize.height;
cellWidth = width / 8;
cellHeight = height / 8;
cellsAcross = width / cellWidth;
cellsDown = height / cellHeight;
pixels = new int[cellWidth * cellHeight];
prevCells = new int[cellsAcross][cellsDown][cellWidth * cellHeight];
//out.writeInt(-123);
out.writeInt(cellWidth);
out.writeInt(cellHeight);
out.writeInt(cellsAcross);
out.writeInt(cellsDown);
out.flush();/***** modified/added here ********/
System.out.println("cellWidth="+cellWidth+" cellHeight"+cellHeight+
" cellsAcross="+cellsAcross+" cellsDown="+cellsDown);
setPriority(Thread.MIN_PRIORITY);
// start();
} catch (Exception e) {
e.printStackTrace();
}
}
boolean keepRunning = true,go=true;
public void run() {
try {
while (keepRunning) {
long start = new Date().getTime();
image = robot.createScreenCapture(new Rectangle(width, height));
// get the underlying int array of pixel data from the image...
// (if this breaks, revert to the PixelGrabber code below)
Raster ras = ((BufferedImage) image).getData();
DataBufferInt db = (DataBufferInt) ras.getDataBuffer();
int[] data = db.getData();
System.out.print("\nSending ");
// process all cells in column order within rows...
for (int j = 0; j < height / cellHeight; j++) {
for (int i = 0; i < cellsAcross; i++) {
int pixelIndex = 0;
int cellOffset = i * cellWidth + j * cellHeight * width;
for (int y = 0; y < cellHeight; y++) {
int rowOffset = cellOffset+ y * width;
for (int x = 0; x < cellWidth; x++) {
pixels[pixelIndex++] = data[rowOffset + x ];
}
}
// see if this cell has changed since last time...
int[] prevPixels = prevCells[i][j];
boolean cellChanged = false;
for (int k = 0; k < pixels.length; k++) {
if (pixels[k] != prevPixels[k]) {
prevPixels[k] = pixels[k];
cellChanged = true;
}
}
// send changed cell to client
if (cellChanged) {
int[] temp = RLEencode(pixels);
out.writeInt(i);
out.writeInt(j);
out.writeInt(temp.length);
for (int word : temp) {
out.writeInt(word);
}
}
}
}//System.out.print("flushing it away....");
out.flush();
System.out.println("Updated in " + (new Date().getTime() - start)
+ " mSec");
Thread.sleep(10);
}
out.writeInt(-1); // EOF
out.flush();
} catch (SocketException e) {
System.out.println("\nScreen viewing Client disconnected");
} catch (Exception e) {
e.printStackTrace();
}
}
int[] RLEencode(int[] data) {
// Find repeated int values in array - replace with 1 value + repeat count
// 1st byte of ints are Alpha - not used here, so that's where
// the count is stored.
int[] result = new int[data.length];
int first = 0, count = 0, index = 0 ;
for (int i = 0; i < data.length; i++) {
if (data[first] == data[i] && count < 126) {
count++;
} else {
result[index++] = (data[first] & 0x00FFFFFF) | (count<<24);
first = i;
count = 1;
}
}
result[index++] = (data[first] & 0x00FFFFFF) | (count<<24);
System.out.println(" Compressed " + // (index-1) + " of " + data.length +
"(" + index*100/data.length + "%)");
data = new int[index];
System.arraycopy(result,0,data, 0, index);
return data;
}
}The thread that takes the incoming data from sender: first time it takes the cell info, then the data sent by sender thread.
public class incomingImg extends Thread{
chatClient cc;
private int opt;
boolean firsttime=true;
int cnt=0;
public incomingImg(chatClient c) throws AWTException {
this.cc = c;
socket = cc.clientSocket;
in = cc.is_img; ////
}
////////
Socket socket;///
DataInputStream in;///
int cellWidth;
int cellHeight;
int cellsAcross;
int cellsDown;
JLabel labels[][];
int[] pixels;
ImageProducer ip;
/////////
void RLEdecode(int[] data, int[] target) {
int index = 0;
for (int i = 0; i < data.length; i++) {
int value = data[i] | 0xFF000000; // reinstate Alpha value 255
int count = data[i] >>> 24;
// System.out.println(index + " = " + count);
for (int k = 0; k< count; k++) {
target[index++] = value;
}
}
}
int count=0;
@Override
@SuppressWarnings("empty-statement")
public void run() {
try {
/* cc.os_img.writeInt(222);
cc.os_img.flush();*/
while (cc.is_img != null && (opt = cc.is_img.readInt()) != -10000) {
count++;
if (firsttime) {
System.out.println("count="+count);
try {
/*
in = new DataInputStream(new BufferedInputStream(socket
.getInputStream()));*/
// cellWidth = in.readInt();
cellWidth = opt;
cellHeight = in.readInt();
cellsAcross = in.readInt();
cellsDown = in.readInt();
System.out.println("cells:" + cellWidth + "x" + cellHeight);
labels = new JLabel[cellsAcross][cellsDown];
pixels = new int[cellWidth * cellHeight];
ip = new MemoryImageSource(cellWidth, cellHeight, pixels, 0, cellWidth);
JPanel panel = new JPanel();
panel.setLayout(new GridLayout(cellsDown, cellsAcross, 0,0));
cc.img_panel.setLayout(new FlowLayout());
cc.img_panel.add(panel);
Image initialCellImage = Toolkit.getDefaultToolkit().createImage(ip);
for (int j = 0; j < cellsDown; j++) {
for (int i = 0; i < cellsAcross; i++) {
labels[i][j] = new JLabel();
labels[i][j].setIcon(new ImageIcon(initialCellImage));
panel.add(labels[i][j]);
}
}
cc.img_panel.validate();
} catch (IOException ex) {
Logger.getLogger(incomingImg.class.getName()).log(Level.SEVERE, null, ex);
}
firsttime = false;
}
else
{ System.out.println("count="+count);
System.out.println("entered, opt="+opt);
// int i = in.readInt();
int i=opt;
/*
if(!firsttime) i = opt;
else i = in.readInt();*/
if (i < 0) {
break;
}
int j = in.readInt();
System.out.println("i=" + i + " j=" + j);
int length = in.readInt();
System.out.println("" + length);
int[] temp = new int[length];
for (int k = 0; k < length; k++) {
temp[k] = in.readInt();
}
RLEdecode(temp, pixels);
// convert array back to image:
Image cell = Toolkit.getDefaultToolkit().createImage(ip);
labels[i][j].setIcon(new ImageIcon(cell));
}
}
} catch (IOException ex) {
Logger.getLogger(incomingImg.class.getName()).log(Level.SEVERE, null, ex);
}
}
}If you put in the stream all the cells then flush it, at the destination it is read one by one... updated cells one by one... or its me not understanding the stream queue :)... thx.
If you put in the stream all the cells then flush it, at the destination it is read one by one... updated cells one by one... or its me not understanding the stream queue :)... thx.
yes definitely,,,,
every cell might processing 0.5 sec...sow when it will take and divide
and process each cell is causing more delay...
instead of image processing do you know any TCP optimization technique....:-/
yes definitely,,,,
every cell might processing 0.5 sec...sow when it will take and divide
and process each cell is causing more delay...instead of image processing do you know any TCP optimization technique....:-/
Well... I ask James about this, maybe it's just me having this delay. Let him tell us the delay he has in LAN (how fast all of the image comes). musthafa.aj, it would be better for you to try some ideas... other then compressing the image. James works with pixel data. I thinks compressing the buffer is a very good idea... it's just the TCP issue and how we read the data. The less length of the data stream queue the less delay we will have. Another idea is to work with image extracting rather than data pixel. Think about that musthafa.aj. +another thing. Don't stick so much with the idea of TCP. This is the first implementation. After this will work (if it will :) ) I propose to try UDP. Faster and realtime. Till then. I want to see what James says about this delay. Maybe it's just me not embedding his codes into mine as expected.
Hi guys.
Don't know why yu get these delays - I'm seeing 200mSec to scan the whole screen and then just the transmission delays for a few changed cells (64*64*4 bytes each, less 30% for RLE compression) = 11k/changed cell.
Across a WiFi link with a 2-second refresh rate it's almost like watching the source screen, and server CPU is < 10%.
I've stuffed the code together with just enough context/infrastructure to run (I'm doing this within quite a large context, all of which is irrelevant here).
The zip contains a server jar and a viewer jar.
The server has no UI, and exits when the client disconnects. If you can't connect in the first place you'll have to crash it from taskmanger or whatever.
The client attempts to re-connect to a server at whatever address it previously found it, and prompts for a host name / IP address if it can't connect (eg first time run).
Let's see how that works for you!
well james....huge thanks for you...
but sorry i just confused because...
you connect two local machine within LAN...
it working slower than i have even at LAN
(2 or 3 seconds...)..
when it comes to over internet it will consumed lot...
that's why i just worried about speed...
now i have one it working faster than yours at LAN...
we worried at INTERNET performance...
Well, being in LAN, I mean connected to myself it's 2 second delay... big for LAN, I mean I want to make it as realtime as possible and come closer to TeamViewer as posiible (event TV uses API and works only on windows). Here is a demo if my project. Try it.. it might take more CPU... but tell me what statistics you got. This application sends compressed JPEG over LAN and internet using TCP socket. It sends the whole image... Just use connect at that IP and port at Client and start at Server... other buttons are some tests and might crush the application. Tell me what you think... You can use mouse and keyboard :). Have some fun too. However the image and refresh rate part is in the interest of this thread.
it working slower than i have even at LAN
(2 or 3 seconds...)..
we worried at INTERNET performance...
It's set to send an incremental refresh every 2 secs (compromise between responsiveness and server CPU loading). That's configurable. At that speed it should be using <10% CPU, and giving a negligable LAN network load.
I understand the internet issue. A full screen image, with lossless compression, is still going to be near a megabyte, and on a 2 megabit ADSL line that's an unavoidable 4 seconds!
All you can do is to minimise the number of bytes transmitted, which is what I've tried to do, without corrupting the screen image with lossy encoding like JPEG.
ps: Just tried without the 2 second refresh rate limit (ie target rate 0), gives me 5 frames/second (200 mSec per refresh), LAN still nowhere near saturated.
to clawsy..when i click start button it will be idle nothing happen..why?
any thing i need to configure in system..;)
Here's a jar without any frame rate limiting - this will send updates as fast as it can. See how that goes?
awesome james...
i taken it into my account..
i will make it for my app suit..
and i will reply...
but clawsy demo not worked..
later i will send my part check it guyz..
Now I understand James... but why does my code (previos, from you) worked so slow? I dont' undestand... did you change somthing in this last demo? This demo is indeed fast.
To musthafa.aj: Start the server and click start. The it will freeze until you start the client and connect it at the IP of the server (127.0.0.1 for local) and at the same port. Then click the Connect button.
To musthafa.aj: Start the server and click start. The it will freeze until you start the client and connect it at the IP of the server (127.0.0.1 for local) and at the same port. Then click the Connect button.
ok i think you done what exactly i was..
ok still we force to make what actually we need and i also try it under my apps that what james suggest...
asap i come with good....
I get this output from the sender:
Sending
Compressed (9%)
Compressed (1%)
Compressed (1%)
Compressed (1%)
Compressed (8%)
Compressed (10%)
Compressed (4%)
Compressed (13%)
Compressed (4%)
Compressed (0%)
Compressed (0%)
Compressed (0%)
Compressed (0%)
Compressed (13%)
Compressed (1%)
Compressed (11%)
Compressed (7%)
Compressed (2%)
Compressed (0%)
Compressed (1%)
Compressed (3%)
Compressed (12%)
Compressed (1%)
Compressed (11%)
Compressed (5%)
Compressed (1%)
Compressed (0%)
Compressed (0%)
Compressed (0%)
Compressed (16%)
Compressed (1%)
Compressed (11%)
Compressed (5%)
Compressed (1%)
Compressed (0%)
Compressed (0%)
Compressed (0%)
Compressed (8%)
Compressed (0%)
Compressed (11%)
Compressed (5%)
Compressed (1%)
Compressed (0%)
Compressed (0%)
Compressed (0%)
Compressed (8%)
Compressed (0%)
Compressed (11%)
Compressed (13%)
Compressed (13%)
Compressed (3%)
Compressed (1%)
Compressed (0%)
Compressed (2%)
Compressed (0%)
Compressed (8%)
Compressed (28%)
Compressed (11%)
Compressed (4%)
Compressed (3%)
Compressed (3%)
Compressed (2%)
Compressed (2%)
Compressed (17%)
Updated in 14813 mSec
... So I gues the sender is the problem. I modified the sleep from 3000 to 10... the sender code is in one of my previous posts...
did you change somthing in this last demo? This demo is indeed fast.
Only change is that I took out the sleep(...) waits that I used to throttle the previous version back to one update every 2 seconds.
i taken it into my account..
i will make it for my app suit..
musthafa: You have my permission to use my code in your application, but you must place a comment
// uses code provided by James Cherrill 2010
in your program near where my code is used.
Cheers
James
I get this output from the sender:
Sending
Compressed (9%)
...
Compressed (17%)
Updated in 14813 mSec
Looks like an unbuffered output stream to me! I write one word at a time, so unbuffered writes are a disaster.
I'll get rid of the compression stats output now, they have served their purpose.
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.