Robot screen capture - heap OutOfMemory while reducing quality

[Vaspar]

I am using Robot's screenCapture method which returns BufferedImage, and then send using Socket to client on Android. If I don't reduce Quality then It works fine but it is very slow as Image is large and takes much time to send.
I am trying to Reduce quality of BufferedImage(using ImageResizer Class), It works but for very limited time and then crash on server end.... It gives me following exception

Exception in thread "Thread-7" Exception in thread "Thread-8" java.lang.OutOfMemoryError: Java heap space
    at sun.awt.windows.WRobotPeer.getRGBPixels(WRobotPeer.java:64)
    at java.awt.Robot.createScreenCapture(Robot.java:444)
    at SystemController.screenCap(SystemController.java:71)
    at Server$ImageServer.run(Server.java:356)
    at java.lang.Thread.run(Thread.java:722)

I am using following class for resizing

public class ImageResizer {
    public static ByteArrayOutputStream resizeImage(BufferedImage image) {
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        try {
            float quality = 0.3f; // Change this as needed
            // get all image writers for JPG format
            Iterator<ImageWriter> writers = ImageIO
                    .getImageWritersByFormatName("jpg");
            if (!writers.hasNext()) {
                throw new IllegalStateException("No writers found");
            }
            ImageWriter writer = writers.next();
            //ImageOutputStream ios = ImageIO.createImageOutputStream(os);
            try (ImageOutputStream ios = ImageIO.createImageOutputStream(out)) {
                writer.setOutput(ios);
                // set compression quality
                ImageWriteParam param = writer.getDefaultWriteParam();
                param.setCompressionMode(ImageWriteParam.MODE_EXPLICIT);
                param.setCompressionQuality(quality);
                writer.write(null, new IIOImage(image, null, null), param);
            }
        } catch (IOException ex) {
            ex.printStackTrace();
        }
        return out;
    }
}

And this method is used for capturing

public ByteArrayOutputStream screenCap(Rectangle captureSize, int i) {
            robot = new Robot();
            screenImage = robot.createScreenCapture(captureSize);
            return ImageResizer.resizeImage(screenImage);
        }

What am I doing wrong?

[JamesCherrill]

I see you creating a new output stream each time, but not closing it - maybe they are not being garbage collected?

You may also find this thread interesting - a few of us were trying to optimise compressing a stream of real-time screen images to send over a socket connection, but with lossless compression (no JPEG artifacts etc). Eventually we got pretty good results by only sending changed pixels each time, and using a simple RLE encode for the pixels we did send.
I stll have the final version in daily use here, and will be happy to share if you're interested.

[Vaspar]

I have updated my code.. and it is working fine now
problem was created by ImageWriter writer;
i was not disposing it
now it is working fine

but your approach is much better by finding difference between two images and sending that
but i have limited time to submit my final year project
how much time would it require to make all changes on client and server end
i am sending image using following code

byte[] imageData = null;
                        byte[] bytesOut = bot.screenCap(new Rectangle(panRect.x, panRect.y, panRect.width, panRect.height), 1);
                        imageData = Base64.encodeBase64(bytesOut);
                        ot.writeLong(imageData.length);
                        ot.write(imageData);
                        ot.flush();

and reading byte array at client end then decode it and convert it in Bitmap (android)

[JamesCherrill]

If your time is limited then it's probably better to spend it making sure what you have is 100%, rather than starting a major upgrade. It's probably a day's work to fit our code and test thoroughly, but that doesn't allow for understanding the compression algorithm and data packing, which could cause you problems if you get asked questions about what you submit...

[bibiki]

...however, I would love to see the code JamesCherrill. Not only because it is code that I would like to make use of, but also to see your code as I am sure I'd learn something since I have been getting quite some help from you last 3 years. thanks in advance.

[JamesCherrill]

OK, It's embedded in a much larger app, so I'll need to identify the right methods to publish, and add a bit of documentation, but I'll do that in the next day or two and post it here for everyone's amusement.

[bibiki]

thanks JamesCherrill. I am enthusiastically looking forward to it.

[JamesCherrill]

OK part 1
This is the method that gets things started... it assumes you already have a socket connection betrween the client and server.

  ScreenServer(Socket clientSocket, int targetRefreshInterval) {
      // targetRefreshInterval is the desired interval (in mSecs) between
      // starting refreshes. (0 means continuous updates.)
      // Actual rate is not guaranteed.
      // Eg targetRefreshInterval = 2000 means try to refresh every 2 seconds

      try {

         robot = new Robot();
         robot.setAutoWaitForIdle(true);
         Dimension screenSize = Toolkit.getDefaultToolkit().getScreenSize();
         width = screenSize.width;
         height = screenSize.height;

         System.out.println("Remote screenviewer connected, refresh interval "
               + targetRefreshInterval + "mSec");

         out = new DataOutputStream(new BufferedOutputStream(clientSocket
               .getOutputStream()));
         out.writeInt(width);
         out.writeInt(height);

         this.targetRefreshInterval = targetRefreshInterval;
         setPriority(Thread.MIN_PRIORITY);
         start(); // this class extends Thread
      } catch (Exception e) {
         e.printStackTrace();
      }
   }

and here's the run method that loops managing the screen cature and sending

  @Override
   public void run() {
      int bytesSent;
      try {
         while (keepRunning) {
            long startTime = new Date().getTime();

            bytesSent = sendScreen();
            if (bytesSent < 0) break; // error

            long endTime = new Date().getTime();
            // System.out.println("Updated " + (bytesSent + 1023) / 1024
            // + "kB, in " + (endTime - startTime) + " mSec");

            long timeToSleep = targetRefreshInterval - (endTime - startTime);
            if (timeToSleep > 0) Thread.sleep(timeToSleep);
         }
         out.writeInt(-1); // EOF code sent to client
         out.close();
      } catch (IOException e) {
         System.out.println("Socket unavailable");
         // e.printStackTrace();
      } catch (Exception e) {
         e.printStackTrace();
      }
      System.out.println("Remote screenviewer disconnected");
   }

... to be continued...

[JamesCherrill]

Part 2
These two methods are the important part of the server
First - get screen images

int sendScreen() throws IOException {
      BufferedImage image = robot.createScreenCapture(new Rectangle(width, height));
      Raster ras =  image.getData();
      DataBufferInt db = (DataBufferInt) ras.getDataBuffer();
      int[] data = db.getData();

      int bytesSent = sendIncrementalRLE(data, prevData, out);
      if (bytesSent < 0) return -1;
      // error
      prevData = data;
      out.flush();
      return bytesSent;
   }

and, at last the encoding/compression

 int sendIncrementalRLE(int[] data, int[] prevData, DataOutputStream out) {

      // Test for unchanged pixels, and just send how many they are, otherwize
      // find repeated int values in array - replace with 1 value + repeat count
      // 1st byte of ints are Alpha - not used here, so that's where
      // the count is stored. Returns no of bytes sent, -1 for error.
      //
      // Output stream format is sequence of 1 integer records describing the
      // data array in natural order (element 0 ... data.length-1)
      //
      // EOF record - int value -1 (FFFFFFFF)
      //
      // Unchanged record (sequence of >=1 unchanged pixels):
      // bits 0-7 are all 0,
      // bits 8-31 are number of consecutive unchanged pixels.
      //
      // Changed record (sequence of >=1 identical changed pixels):
      // bits 0-7 are number of consecutive pixels the same 1-254,
      //          (255 is not available because of clash with EOF)
      // bits 8-31 are the 3 byte RGB values for these pixels.
      //
      // Skipping unchanged pixels is based on a transparent pixel idea
      // from DaniWeb user Clawsy. Thanks Clawsy.

      try {
         int bytesSent = 0;
         int i = 0; // index into data array
         int equalCount = 0, dataValue = 0, dataCount = 0;
         while (i < data.length) {
            // count number of unchanged pixels...
            while (prevData != null && i < data.length && 
                  equalCount < 0xFFFFFF  && data[i] == prevData[i]) {
               equalCount++;
               i++;
            }
            if (equalCount > 0) {
               out.writeInt(equalCount);
               bytesSent += 4;
               equalCount = 0;
            }
            if (i >= data.length) break;
            // count number of consecutive identical pixels...
            dataValue = data[i];
            dataCount = 1;
            i++;
            while (i < data.length && data[i] == dataValue && dataCount < 254) {
               // dataCount cannot be 255 because that can look like an EOF
               // (255 pixels of pure white are FFFFFFFF, which = -1, = EOF)
               dataCount++;
               i++;
            }
            out.writeInt((dataValue & 0x00FFFFFF) | (dataCount << 24));
            bytesSent += 4;
         }
         return bytesSent;
      } catch (IOException e) {
         return -1; // error
      }
   }

... to be continued ...

[JamesCherrill]

Part 3 (last) the client's main run mthod

 public void run() {

      try {
         in = new DataInputStream(new BufferedInputStream(socket.getInputStream()));
         width = in.readInt();
         height = in.readInt();
         System.out.println("Screen Watcher connected to Server. Screen size " + width
               + "x" + height);
         out = new DataOutputStream(new BufferedOutputStream(socket.getOutputStream()));
      } catch (Exception e) {
         e.printStackTrace();
      }

      // create in-memory Image, and get access to its raw pixel data array
      image = new BufferedImage(width, height, BufferedImage.TYPE_INT_ARGB);
      DataBufferInt db = (DataBufferInt) image.getRaster().getDataBuffer();
      int[] pixels = db.getData();

      JFrame frame = new JFrame("Remote Screen");
      frame.setLocation(20, 20);
      JLabel label = new ImageLabel();
      label.setIcon(new ImageIcon(image));
      frame.add(new JScrollPane(label));
      frame.pack();
      setWindowSize(frame);
      frame.setVisible(true);

      // frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
      frame.addWindowListener(new WindowAdapter() {
         public void windowClosing(WindowEvent e) {
            shutdown();
         }
      });

      try {
         while (!shuttingDown) {
            // read updates as long as they keep coming...
            int i = 0; // index into pixel array
            while (i < pixels.length) {
               int next = in.readInt();
               if (next == -1) { // EOF from server
                  System.out.println("EOF received from server");
                  shuttingDown = true;
                  break;
               }
               if ((next & 0xFF000000) == 0) { // "Unchanged" record
                  // skip specified number of unchanged pixels
                  i += next;
               } else { // "Changed" record
                  int value = next | 0xFF000000; // RGB, with Alpha byte 255
                  int count = next >>> 24; // number of repeated values
                  for (int k = 0; k < count; k++) {
                     pixels[i++] = value;
                  }
               }
            }
            label.repaint(); 
         }
      } catch (Exception e) {
         if (shuttingDown) {
            // ignore this - applications is terminating and socket was closed.
         } else {
            e.printStackTrace();
         }
      }
   }

and finally a little extension of JLabel to provide optimised re-paints from an image to display the screen

    class ImageLabel extends JLabel {
      // repaints directly from received BufferedImage
      private static final long serialVersionUID = 1L;

      public void paint(Graphics g) {
         g.drawImage(image, 0, 0, null);
      }
   }

I think that's all the interesting bits. You may find some refs to other stuff, but they are probably not relevant.
Have fun
J

[Vaspar]

Thanks alot James :)
My teacher has extended date of Project demonstration.... I have almost month now to improve what I have done till now :)
I will integrate your code in my Project and will add this thread as reference :)

Thanks alot for your help

[bibiki]

hey JamesCherrill, thanks a lot for your code. I'll look at it closely. It is very much appreciated.

[JamesCherrill]

Don't look too closely! See previos thread - it was an experimental work in progress, not a fully designed thing. It's only the actual compression/decompression that's coded properly.