@rptoffitt ok will try to learn and adapt as much as you have suggested here. Hopefully its helpful.

@rproffitt I mean we want to do some kind of simulation to try to understand the actual situation. We dont want to be caught up with the real situation and think that moment is bad. I dont get you here "Still wonder why you went down that road since it tends to be costly. That is, if you are competing with another company they would have a price advantage" Which road mean ? I dotn get you here?

@rproffitt well based on the googling result I think best is to break it down to multiple server right. So any other tips you would like to add on ?

@rproffitt what kind of issue will I ran into. Basically I am going to have multiple communication server not just one single server to handle the whole tcp connection. How will that be helpful e.g. each server only server up to 5K devices. So I am not sure what type of hardware specs is needed. I want to know roughly on this and also bandwidth. These are the things I am on the look out now.

@rprofitt yes I know there many prior to handling multiple connections e.g. bandwidth, servers etc. Yes I will be willing to talk to your contact or to you on personal email. For now I just want to do a load test and throughput test to find what will be exact server requirements when we reach that traffic and scale. So that is what to be exact I am trying to learn and understand.

I told you this not a team of one. We have designer, programmer, map guys, server guys etc. So definitely not a single person running is crazy no way to achieve it either.

@rproffitt seriously I am still lost on the understanding of the sequential part when you say that at the socket.accept and how they get it running in multithreaded. The issue I am still not clear is the testing how to craft up the test still lost at these 2 points.

I agree this not a single client exactly is more like a semi govt application which has many mini customer and later we need to do some big data stuff on this data. That is the reason we need one single central database.

@rpfroffitt yes I know tcp is kind of more costly cause we need to setup the communication etc. Before going further on testing I want know more the sequential buffering for tcp I am not so clear on that? Can you bried me more on that kind of confuse?

I know about testing need tips what and how to test? Kind of very lost here?

@rproffitt ok I got it unfortunately the device itself support tcp only so not much of choice left here. No problem I wont ask you about your udp. I want to first understand so you said every tcp which comes in is buffered rite. Then it pick up the socket.accept and create a new socket thread is that correct? There after the device is connected and send the data over the socket which have been build ? Am I right here? So thereafter the socket is build is it still sequentially done right. So how I do a stress test on how many thousand of socket it can handle in one second or one minute to know the throughput?

@rprofitt actually I am trying to get unstuck from the parallel processing view. Let me understand at the lowest level say now I got 10 devices trying to make connection to my socket server. So what is happening is that all the data is line up in the buffer right. So then how is the socket start here.

Socket socketConn1 = serverSocketConn.accept();
                                    new Thread(new ConnectionHandler(socketConn1)).start();    

Can I say that is this done in sequential method? So is it safe for the device to keep holding the socket and not close for a long time or is better to close?

Actually the monster machine we have not setup we are planning to setup not yet on the cloud too.

Can you give me more tips on this " Get a measure of throughput. How many transactions per second can you achieve. " How get this metrics any method or can I work based on my stress test script ? Please give more hints as I am willing to learn and change accordingly.

@rproffitt ok I will go with your advice. So in other words everything is buffered at the socket level is that what your trying to explain to me? So how to test for this "How many requests per second can my system process?" What about the code snippet which I have put some to do the stress test? So if its trully not parallel then is all sequential why do we need multithread just would like to understand moer details?

@rpfrofitt Wow this something new to me. I thought all the while the data is concurrent at the data center which what you said makes sense though.

How can I create multiple client via java I have this code snippet to do a stress test. The problem this is a sequential test how to make it simulate as parallel test?

public class stress1 extends Thread {
public static void main(String[] argv) {
    try {
        for (int i = 0; i < 50; i++) {
        Socket socket = new Socket("192.168.2.102", 8000);
        PrintWriter out = new PrintWriter(socket.getOutputStream(), true);
        BufferedReader in = new BufferedReader(new InputStreamReader(
        socket.getInputStream()));
        out.println("$ACTEST35,30061104075130528955N10024852E000068*03A1*");
        System.out.println(in.readLine() + i);
        out.close();
        in.close();
        socket.close();
        }
    } catch (Exception e) {
        e.printStackTrace();
   }
  }
}

@cereal thank you I have been reading about it. The problem is that I cant send the same data for the 50K connection which I need to make I need to simulate as different device id for that purpose can mysqlslap handle this?

@rprofitt did you face slow down while insert multiple concurrent data which already have hundred of millions of data. Any idea or experience handling those traffic ?We are a team not just alone programmer. The issue now we are looking into solution for this as we anticipate the growth in near future we dont want to be caught later and not knowing that to do when we are facing the traffic so we are doing research and study from now onwards. I dont get you when you said you crushed the ram and cpu you mean its too much for the ram?

@rprofitt I wonder how to do a parallel 50 thousand load testing ? What tool is available but if I write my own socket client that will be sequential not parallel. If you notice at the tcp communicator I will be doing a number select and updates and finally only do the insert. So in one minute can it handle up to nearly 50 thousand of these type of select,update and insert queries? How many thousands you had and what was your hardware specs? I am sure I will need to do some tweak at the db level like innodb tweak and query indexing ? Over time the db will increase its size and insert will slow down right?

@rproffitt sorry for my earlier post I got some typo error. Let me share below is how my tcp communicator snippet code looks like. Tcp communicator decodes the GPS location and inserts that data into the database and before inserting we do some selects and updates but all I do it using prepared statements. Right now, one thread of TCP communicator serves one device request.Immediately after creating the thread we get one connection from the pool. After decoding the GPS data is where we perform the multiple select, update and insert for each data. As number of devices are increasing, the number of concurrent connections to our Mysql database are also increasing. We are now anticipating 30 to 50 thousand devices pumping data every minute.Currently below is how the snippet of the whole tcp communicator looks like. I know eventually we will facing both insert bottleneck into the database. What will the best solution to over come this scenario? Will Java also be able to handle this many concurrency ?

public class comm8888 {
    HikariDataSource connectionPool = null;
    private Socket receivedSocketConn1;
    ConnectionHandler(Socket receivedSocketConn1) {
      this.receivedSocketConn1=receivedSocketConn1;
    }
    Connection dbconn = null;
    public void run() { // etc
     DataOutputStream w = null;
     DataInputStream r = null;  
     String message="";
     receivedSocketConn1.setSoTimeout(60000);
       dbconn = connectionPool.getConnection();
     dbconn.setAutoCommit(false);
     try {
         w = new DataOutputStream(new BufferedOutputStream(receivedSocketConn1.getOutputStream()));
         r = new DataInputStream(new BufferedInputStream(receivedSocketConn1.getInputStream()));
         while ((m=r.read()) != -1){
             //multiple prepared based sql select,update and insert here.
         }
     }
     finally{
        try {
            if ( dbconn != null ) {
              dbconn.close();
            }
        } ...

@rprofitt. What I mean is my how I collect the gps data is via a java application. In the java application I do connection pooling and prepared statement. That is what I mean so for no clarification. Should I put my code snippet?

What java tios will you give I am doing connection pooling n prepared statement. What else will you help out based on your experience?

Yes I mean gps device. Our interval will be one minute data. The comm server will be java. I plan to have multiple comm server. I have tried so far with around 2K devices. So i want to prepare now for bigger traffic. What I am worry can mysql handle this traffic?

Hi All,
I am having a system which receive data from around 30K gps . Each unit will be inserting concurrently around 1 minute.
1.What tweak should I add to cater for this? I target my ram to be 48Gb with 24 cpus.

  1. Once the data grow should I do partition or leave it . I am worried as the data grow the insert will be slow?
  2. What should I anticipate and handling for heavy traffic?
  3. I plan to have replication data so that the reading can be done from the slave? But what I am worried is that if I delete data from the master then it will effect slave too right?
  4. Is possible to delete the partition from master but not from the slave?

I have the following socket connection codes. Where for every connection is one new thread. After receiving the data which is done via the producer then I put into a queue for next database processor which is the consumer. This works fine but the moment the queue gets build up it can hang or even take even hours to clear cause is like so many producers and just one single consumer. I am thinking is it possible for me to have one consumer handles one producer meaning each message goes into its own queue gets process and that it. What changes can I do to handle it or must I build a thread pooling ?

public class cServer {
   private LinkedBlockingQueue<String> databaseQueue = new LinkedBlockingQueue<String>();
   class ConnectionHandler implements Runnable {

    private Socket receivedSocketConn1;   
    ConnectionHandler(Socket receivedSocketConn1) {
      this.receivedSocketConn1=receivedSocketConn1;
    }     
      // gets data from an inbound connection and queues it for databse update
      public void run() { // etc

         // you already have most of this in your existing code.
         // it uses the shared databaseQueue variable to queue message Strings         
         BufferedWriter w = null;
         BufferedReader r = null;      
          String message="";
          try {

             PrintStream out = System.out; 
             BufferedWriter fout = null;
             w =  new BufferedWriter(new OutputStreamWriter(receivedSocketConn1.getOutputStream()));
             r = new BufferedReader(new InputStreamReader(receivedSocketConn1.getInputStream()));

             int m = 0, count=0;
             int nextChar=0;

             while ((nextChar=r.read()) != -1) 
             {

                  message += (char) nextChar;  

                  if (nextChar == '*')
                  {

                     databaseQueue.add(message);  
                     message="";                
                  }
                 }
              } 
              catch (IOException ex)  
              { 
                   System.out.println("MyError:IOException has been caught in in the main first try");
                   ex.printStackTrace(System.out); ...

Hi Taywin,

                 Actually I did not put my full codes because its too lengthy. Below is how I declare my connectionPool.

 public class sp8888 {
  BoneCP connectionPool = null;

So this is why I can use in the run function and call the dbconn1 = connectionPool.getConnection();. So what else could be wrong? Could it be that the bonecp is leaking how to confirm on that?

I have done a socket programming using bonecp as the database pooling for mysql. Suddenly I notice a lot of socket connection made but then after a few hours I got too many open files. I have set the timeout but somehow it did not timeout I don't know why. Why did the timeout did not trigger or worked in this scenario? I need to know is this due to database pooling or socket problem?
Below is snippet of my code. In my code I notice in the logs the connection is made and stops at this line-

System.out.println("\n\n Trying establish a new db connection ");
dbconn1 = connectionPool.getConnection(); 

There is possiblity it does not get the connection from pool but there is no error or any socket connection timeout. Is there any solution for this? How over come this or resolve to that is the pool is full to throw some exception.

 private Socket receivedSocketConn1;
ConnectionHandler(Socket receivedSocketConn1) {
  this.receivedSocketConn1=receivedSocketConn1;
}
BufferedWriter writer1 = null; 
 Connection dbconn1 = null;     
     public void run() { // etc
     writer1 = null;
     String message="";
     BufferedReader reader1 = null; 
     try { 

         writer1 =  new BufferedWriter(new OutputStreamWriter(receivedSocketConn1.getOutputStream()));
         reader1 = new BufferedReader(new InputStreamReader(receivedSocketConn1.getInputStream()));
         receivedSocketConn1.setSoTimeout(60000);
         int nextChar=0;
         while ((nextChar=reader1.read()) != -1) {    
             message += (char) nextChar; 
             if (nextChar == '*'){

                 try{   
                    System.out.println("\n\n Trying establish a new db connection ");
                    dbconn1 = connectionPool.getConnection();
                    dbconn1.setAutoCommit(false);
                    System.out.println("\n\n Checking db connection status "+dbconn1.isClosed());
                    if ((dbconn1 == null) || dbconn1.isClosed()) {
                         System.out.println("\n\n db connection status is closed");
                         dbconn1 = connectionPool.getConnection();
                         dbconn1.setAutoCommit(false);
//other codes ...

I have the following snippet of codes first each of the thread is a socket waitign for connection where its gets a connection does all he necessary. Thereafter another section in the run it does this //run a new db connection and does a select where all those have not been updated will be send to webservice and later update their status. The weird part I notice at times this section is ran twice? Could it be the previous thread was not closed?

 public void run() {

 try{
      //socket connection codes which does all the necessary and finally close the connection.
 }
          catch (SocketTimeoutException ex)  
          { 
               System.out.println("SocketTimeoutException has been caught in in the main first try");
               ex.printStackTrace();
          }  
          catch (IOException ex)  
          { 
               System.out.println("IOException has been caught in in the main first try");
               ex.printStackTrace();
          }  
          catch (Exception ex)  
          { 
               System.out.println( receivedSocketConn1.getInetAddress() +":"+receivedSocketConn1.getPort()+" IS CONNECTED ");

               System.out.println("MyError:Exception has been caught in in the main first try");
               ex.printStackTrace(System.out);
          }   

   //run a new db connection and does a select where all those have not been updated will be send to webservice and later update their status.

   }

I am working on a small as2 message passing. Below are my codes. First is the host which will send the header and data to the server. Then the server is suppose to form a reply in this format http://www.edidev.net/edidev-ca/help/Using/Using_AS2/Using_MDN/MDNRequestFromAs2SubjMsg.htm

<?php

  $header = "HTTP/1.0\r\n";

$header.= "CONNECTION: Keep-Alive \r\n\r\n"; 
  // Here is the data we will be sending to the service
  $some_data = array(
    'message' =>'Test1', 
    'name' => 'Chad'
  );  

  $curl = curl_init();
  // You can also set the URL you want to communicate with by doing this:
  // $curl = curl_init('http://localhost/echoservice');

  // We POST the data
  curl_setopt($curl, CURLOPT_POST, 1);
  // Set the url path we want to call
  curl_setopt($curl, CURLOPT_URL, 'http://localhost/pro1/helloservice.php'); 

  curl_setopt($curl, CURLOPT_HTTPHEADER, array(
    'AS2-FROM: TEst1',
    'X-BLUECOAT-VIA: test1',
    'DISPOSITION-NOTIFICATION-TO: TEst1',
    'MESSAGE-ID: 15610.1493.14155.*****.com',
    'AS2-TO: Test2',
    'CONTENT-LENGTH:5',
    'CONTENT-TYPE: application/edifact',
    'CONTENT-TYPE: application/edifact',
    'HOST: localhost:8080'
    ));

  // Make it so the data coming back is put into a string
  curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
  // Insert the data
  curl_setopt($curl, CURLOPT_POSTFIELDS, $some_data);

  // You can also bunch the above commands into an array if you choose using: curl_setopt_array

  // Send the request
  $result = curl_exec($curl);
  // Free up the resources $curl is using
  $info = curl_getinfo($curl);  
  echo 'content type: ' . $info['content_type'] . '<br>';
  //echo 'http code: ' . $info['http_code'] . '<br>';

  curl_close($curl);

  echo $result;

?>

Below is the codes on the server.

<?php
  //echo 'Your message was: ' . $_REQUEST["message"] . ' and your name is: ' . $_REQUEST["name"];
$headers = apache_request_headers();

foreach ($headers as $header => $value) {
    echo "$header: $value <br />\n";
}
?> ...

I have a very simple php curl application where below is my codes I used to send header and body to server.

<?php

$xml_data = "<Request><NewOrder>";

$URL = "http://localhost/rece1.php";
$header ="";
// Build header as array for cURL option
$header = "HTTP/1.0\r\n";
$header.= "MIME-Version: 1.0\r\n";
//$header.= "Content-type: application/PTI46\r\n";
$header.= "Content-length: ".strlen($xml_data)."\r\n";
$header.= "Content-transfer-encoding: text\r\n";
$header.= "Request-number: 1\r\n";
$header.= "Document-type: Request\r\n";
$header.= "Interface-Version: Test 1.4\r\n";
$header.= "Connection: close \r\n\r\n";              
$header.= $xml_data;   

// Define cURL options, then connect to server while saving response
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL,$URL);

curl_setopt($ch, CURLOPT_HEADER, false); 
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $header);
//curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
//curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$f = fopen('request.txt', 'w');
curl_setopt($ch,CURLOPT_VERBOSE,true);
curl_setopt($ch,CURLOPT_STDERR ,$f);

$rawResponse = curl_exec($ch);  

echo "TEST : ".$rawResponse;           

if (curl_errno($ch)) {
   print curl_error($ch);
} else {
   curl_close($ch);
}

?>

At the server end (rece1.php) I have this.

<?php
$headers = apache_request_headers();

foreach ($headers as $header => $value) {
    echo "$header: $value <br />\n";
}

?>

What I get the results is Empty reply from server. I would like to receive the header information back from server with the body formatted in certain format which some ack message. How can I achieve that?

I agree with you david. Yes the problem now is finding the resource with old and diverse skill and maintaining it. We have already have the idea on how to redevelop the whole software into the new php + java platform. Offcourse the new software will be far more less complex as I notice lots of things from jboss can be eliminated.

Hi,
Currently we have legacy system which is build purely using java and runs on jboss. We notice its using a lot of jms and also httpurl connectivity to pass on files. The system basically supports rossetta net framework where it allows file transfers between trading partners. We notice with the currenty technology we can build the all front end interface and run fully on apache httpd server and leave the back end running on java which deals on http connectivity. Is this the right move for us to proceed or stay put with jboss ?

Dear Dany and Hericles,
                       Yes the template which I am using after doing further study is using bootstrap based. Actually how does fluid works. It is a standard works by itself or we need define it in the css file? Sorry kind of new to these.

I have googling a lot about this debate to move from table to div. What are your advices here to move or not. I have below my codes building layout using div. The issue I find with div for example below codes which I am referring to a template . MY issue is that on different resolution it looks different there is not like fix size. For eg. in some resolution its just nice and some it shrink. What is the best solution to ensure it stay in every resolution which I know if I would go back to table it will work but that is not advisable I guess.

<body>

    <div class="navbar">
        <div class="navbar-inner">
            <div class="container-fluid">
                <a class="btn btn-navbar" data-toggle="collapse" data-target=".top-nav.nav-collapse,.sidebar-nav.nav-collapse">
                    <span class="icon-bar"></span>
                    <span class="icon-bar"></span>
                    <span class="icon-bar"></span>
                </a>
                 <img alt="Logo" src="img/logo20.png" /> 

                <!-- theme selector starts -->
                <div class="btn-group pull-right theme-container" >

                    <h5>User Name:  my demo</h5>
                    Logout
                </div>

            </div>
        </div>
    </div>  

        <div class="container-fluid">
        <div class="row-fluid">

            <!-- left menu starts -->
            <div class="span2 main-menu-span">
                <div class="well nav-collapse sidebar-nav">
                    <ul class="nav nav-tabs nav-stacked main-menu">
                        <li class="nav-header hidden-tablet">Vtst1</li>
                        <li><a class="ajax-link" href="test1.php"><i class="icon-home"></i>  <span class="hidden-tablet">Test1</span></a></li>
                        <li><a class="ajax-link" href="test2.php"><i class="icon-eye-open"></i>  <span class="hidden-tablet">Test2</span></a></li>

                        <li class="nav-header hidden-tablet">Admin</li>
                        <li><a class="ajax-link" href="addM.php"><i class="icon-home"></i>  <span class="hidden-tablet">Master</span></a></li>
                        <li><a class="ajax-link" href="gridU.php"><i class="icon-eye-open"></i>  <span class="hidden-tablet">UD</span></a></li>

                        </ul>

                </div><!--/.well -->
            </div>
                        <!--/span-->
            <!-- left menu ends -->

            <div id="content" class="span10">
            <!-- content starts -->

            <div>
                <ul class="breadcrumb">
                    <li>
                        <a href="#">Home</a> <span class="divider">/</span>
                    </li>
                    <li>
                        <a href="#">Forms</a>
                    </li>
                </ul>
            </div>

            <div class="row-fluid ">
                <div class="box span6">
                    <div class="box-header well" data-original-title>
                        <h2>Add Master</h2>

                    </div>
                    <div class="box-content"> ...
commented: Really ! Helped me out Thanks ! +0

I am linking to a webservice all works fine. Below is my codes.

private static void printSOAPResponse(SOAPMessage soapResponse) throws Exception {
        TransformerFactory transformerFactory = TransformerFactory.newInstance();
        Transformer transformer = transformerFactory.newTransformer();
        Source sourceContent = soapResponse.getSOAPPart().getContent();
        System.out.print("\nResponse SOAP Message = \n");
        StreamResult result = new StreamResult(System.out);     
        transformer.transform(sourceContent, result);
    }

I am not too sure of the transformerfactory. What I need to do now is to traverse through the results and look for below tag.

<Table diffgr:id="Table1" >

and there after there will be few tags in it for e.g.

<rID>1212</rID>
<sNo>15677</sNo>

So what is the best way as some require to covert it into string is that necessary?