There was a thread that covered part of this issue, but it never really got resolved completely.

I now have everything working the way it is supposed to with the single exception of the transfer of the .csv file from my server to the secure server of my vendor.

If anyone could take a few minutes to help me wrap this thing up, I would appreciate it. Was trying not to get into the weekend with out a resolution, but that may not happen.

I'll post the piece of code that relates specifically to this, but if you need the entire script, (only 140 lines) then I can provide that as well.

It is apparently getting logged into the vendors server, based on where the error occurs.
The result that I'm getting is that the transfer fails, and I don't know why.

Any positive feedback and suggestions would be greatly appreciated.

thanks
douglas

    $domain = 'http://www.'.$siteurl;
    $directory = '/enroll_updates/';
    $filename =  'TEST_24houredocs_STANDARD_'.date('Ymd',(strtotime('-1 day',strtotime($created)))).'.csv';

// writes .csv file to my server
    $fd = fopen ( $_SERVER['DOCUMENT_ROOT'].$directory.$filename, "w");// open file to write to
    fputs($fd, $output);// write the data to the opened file
    fclose($fd); // Close the file

    // use CURL to ftp to vendor secure server
    //Create your connection
    $dest_file_path = '/';// to put it in the root directory
    $ftp_conn = ftp_ssl_connect('ftps.vendorurl.com');
    $user='username';
    $pass='password';
    //Login
    $login_result = ftp_login($ftp_conn, $user, $pass);

    if( $login_result ){
      //Set passive mode
      ftp_pasv( $ftp_conn, true );
      // Transfer file
      $transfer_result = ftp_put( $ftp_conn, $dest_file_path, $filename, FTP_BINARY );

      //Verify if transfer was successfully made
      if( $transfer_result){
        $subject="Successful at ";// successful tranfer of data file
      }else{
        $subject="Failed at ";//tried to send file and failed
      }
    }

Recommended Answers

All 24 Replies

Is it even possible to go server to server like what I'm trying to do?

It appears that all references in anyhthing I have found in seaching the web refer to uploading a local file to a remote server.

But I am generating the .csv file on the fly, which is being written to my server, and I need it written to the vendors secure server as well. The only way that I have to do that is via FTP, and short of doing a manual FTP every day, the only way I know of is what I'm trying...

PLEASE, if someone could help me with this, I really need to fing a resolution.

thanks
Douglas

I can't tell you the number of google searches I have done in relation to this project over the past days (seems like weeks)...

Question on the write permissions... is there a difference in doing it this way and just logging in and using FTP?

Because they set up the space for me to ftp my files into, so I would only imagine that they have it set up in such a way as to allow it. That is how they receive files from all their clients. But most of their clients only upload once a month with changes, so they probably do it manually.

I'm doing it daily, which is why I'm trying to automate it.

Thanks for the new link. I will check it out. I would have never imagined this would be such a daunting task.

Douglas

I guess this one is going to be on hold until Monday anyway...

in my rush to get this done, I never even questioned the login that they sent me, but I just tried to log in through SmartFTP, which is what I use all the time for that purpose, and it won't let me log in.

So, thanks for your feedback, but until I can manually log in, I don't imagine that I will be able to remotely log in.

Doug

Member Avatar for diafol

That's a fair call. If manual login fails, I can't see that a programmatic login should succeed.

Just an FYI update

In working with the rep from the vendor company, we finally got me logged in manually and uploaded a file, so I know it works. Why it worked, I don't have a clue.

What it was is this...
Normal address to make a connection using SmartFTP is:

    domainname.com
    username
    password
    port(21)

But to log in with this one was:

    ftps.domainname.com
    username
    password
    port(990) which I had tried with no luck

What I hadn't tried was preceding the domain name with the ftps, because I never preceded the other names with ftp, so never gave it a thought.

I actually also realized that there is another way to define an address to log into, using a form where you can define it as an implicit secure login or an explicit secure login. Turns out it needed to be Implicit.

Don't know what exactly that means, but it worked. So if anyone knows, I'd be interested to learn.

Bottom line - I still can't log in with my script. it fails the log in so I'm having the vendors IT dept look at it and see if they can tell me why, because I don't believe it is one that will be solved here, and I'm afraid I would be pulling my hair out if I continued as I was.

Thank You for trying though. I'll mark it solved just to clear the books.

Douglas

Thanks again for the response...

It seems that I keep going round in circles with this one, and am almost at a point of total frustration. If I thought screaming would help, I'd try that.

Tried the cURL for the 3rd time now with no better results.
I've tried every variation of the $ftp_server url that I can think of and can't seem to get connected to the server.

The vendor determined that I could log on explicit if I used port 21, so I tried that manually, and it worked just fine, but tried it with my script, and got nowhere, just like before.

Wish I could just hire someone to get me past this, and let me move on...

My other alternative is to do a manual FTP upload of the file EVERY DAY... what a PITA that would be.

Any other suggestions or ideas or prayers would be welcome.

thanks
Douglas

Member Avatar for diafol

Forget manual - it never works. You become a slave. Nuh!

It makes me think though that if you can log on manually (explicitly), then you should be able to get it to work programmatically. I know that's little comfort, but would you like to post your current code, so that some of our resident experts can have a look?

Nope, not using a windows machine.

Yea, manual anything is never what I want to do. I automate every little thing that I can.

My thinking was also if I could log in manually, that the rest would be a breeze.. man, was I wrong.

Sure, I'll post the entire script... it is only 140 lines long, and the part that is giving me problems is in the 100 - 125 range I get the $subject line that says 'REMOTE LOGIN FAILED...' Hope the length of the script doesn't scare anyone off from looking at it.

the first 3/4 of the script deals with creating the .csv file and putting it on my server, and all that works just fine...

It is just trying to get it from my server to their server which starts where it says " //Create connection to vendor secure server "

    <?php  // script name vendor_order_gen.php - used to generate .csv file of
    // enrollment records and ftp to secure server.
    include "all_inc/config_site.php";
    include "all_inc/functions.php";
    $created = date("Y-m-d H:i:s", time()) ;
    $Group_ID='11111';
    $SubgroupID='22222';
    $GroupNum='33333';
    $demographics='1';// always set to update Vendor files
    // query to get list of mem_ids from prod_enroll belonging to
    // Primary members that were modified or added yesterday
    $sql = "
      SELECT mem_id
      FROM prod_enroll
      WHERE DATE(last_update) = DATE_ADD(CURDATE(), INTERVAL -1 DAY)
      AND rel_code = 'E'
    ";
    $E_sql = $sql; //  just to send in email for test
    $result=mysql_query($sql);
    $num_rows = mysql_num_rows($result);
    if ($num_rows>0){
      $output='';
      while($row=mysql_fetch_array($result)){ // step through each Primary record
        $mem_id=$row[0];
        $static_1 =$Group_ID.'",'.'"'.$SubgroupID.'",'.'"'.$GroupNum.'",'.'"'.$mem_id;
        // contains GroupID / SubgroupID / GroupNum and / mem_id of primary
        //  this query is for the static information on each member/family to upload
        $sql = "
          SELECT email, addr1, addr2, city, state, zip, phone, plan
          FROM members
          WHERE mem_id = '$mem_id'
          LIMIT 1
        ";
        $result=mysql_query($sql);
        $row=mysql_fetch_array($result);
        //  $static_2 - contains addr1 / addr2 / city / state / zip / phone / email
        $static_2 =$row[1].'",'.'"'.$row[2].'",'.'"'.$row[3].'",'.'"'.$row[4].'",'.'"'.$row[5].'",'.'"'.$row[6].'",'.'"'.$row[0];

        // this is the query to get the variable information
        $sql = "
          SELECT enroll_id, rel_code, fname, lname, gender, birthdate, cover_start, cover_end, enroll_status
          FROM prod_enroll
          WHERE mem_id = '$mem_id'
          AND DATE(last_update) = DATE_ADD(CURDATE(), INTERVAL -1 DAY)
          AND enroll_status!='A'
          ORDER BY enroll_id
        "; // get all NON-archived records for member/dependents
        $result=mysql_query($sql);
        $body_ct=0;

        while($row=mysql_fetch_array($result)){// walk through prod_enroll records
          $fam_mem[$body_ct]=$row;

          // create .csv record.
          // variable_1 contains fname / middle initial field blank / lname /
          $variable_1 =$fam_mem[$body_ct][2].'","'.'","'.$fam_mem[$body_ct][3];

          // variable_2 contains rel_code / dob / gender / start date / end date / demo-always 1
          $variable_2 =$fam_mem[$body_ct][1].'",'.'"'.$fam_mem[$body_ct][5].'",'.'"'.$fam_mem[$body_ct][4].'",'.'"'.$fam_mem[$body_ct][6].'",'.'"'.$fam_mem[$body_ct][7].'",'.'"'.$demographics.'","'.'"';

          $fam_member ='"'.$static_1.'",'.'"'.$variable_1.'",'.'"'.$static_2.'",'.'"'.$variable_2;
          if ($row[1]=='E'){
            $prime_member=$fam_member;
            $fam_mem_ct=1;
          }else{
            ++$fam_mem_ct;
          }

          if (($row[1]=='S') && ($fam_mem_ct!=2)){// include E record before S
            $output .= $prime_member."\n";
          }

          $output .= $fam_member."\n";
          ++$body_ct;
        }
      }
      // Create Domain / Directory / file / and check_file variables
      $domain = 'http://www.'.$siteurl;
      $directory = '/enroll_updates/';
      $filename =  'TEST_24houredocs_STANDARD_'.date('Ymd',(strtotime('-1 day',strtotime($created)))).'.csv';
      $check_file = $domain.$directory.$filename;
      // check to see if this filename has already been submitted
      // if yes - DO NOT resubmit it
      $ch = curl_init("$check_file");
      curl_setopt($ch, CURLOPT_NOBODY, true);
      curl_exec($ch);
      $retcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
      // $retcode >= 400 -> not found, $retcode = 200, found.
      curl_close($ch);
      if($retcode=='200'){ // this file already exists
        // change admin email subject indicating duplicate file being created in error
        $subject="PROCESSING ERROR while ";
        $error_process="<p>THIS FILE HAS ALREADY BEEN PROCESSED AND SUBMITTED<br><br>INVESTIGATE IT</p>";
      }else{// New file name, so send it
        // This creates the .csv file on my server
        // and used for future verification that a daily file has not already been sent.
        $fd = fopen ( $_SERVER['DOCUMENT_ROOT'].$directory.$filename, "w");// open file to write to
        fputs($fd, $output);// write the data to the opened file
        fclose($fd); // Close the file

 //Create connection to vendor secure server
        $dest_file_path = '/';
        $ftp_conn = ftp_ssl_connect('ftps.vendorurl.com',21);
        $user='username';
        $pass='password';
        //Login

        if(ftp_login($ftp_conn, $user, $pass)){
          //Set passive mode
          ftp_pasv( $ftp_conn, true );
          // Transfer file
          $transfer_result = ftp_put( $ftp_conn, $dest_file_path, $filename, FTP_BINARY );
          ftp_close($ftp_conn);
          //Verify if transfer was successfully made
          if( $transfer_result){
            $subject="Successful at ";// successful tranfer of data file
          }else{
            $subject="Failed at ";//tried to send file and failed
          }
        }else{
          $subject="REMOTE LOGIN FAILED when ";//tried to login remotely - failed
        }
      }
    }else{
      $subject="No New Records Available for ";
    }
        print "testing only<br>".$output; // just to display what was in the output variable to test
    // Send email to admin no matter what the results
      $to = $support_mail;
      $subject .= "Generating vendor order";
      $message = $subject."<p>Vender order for yesterday has been generated</p>
        <p>File name is : ".$filename."</p>
        <p>E_sql is : ".$E_sql."</p>
        <p>Here is the content</p>
        <p>".$output."</p>";
      if (isset($error_process)){$message=$error_process.$message;}
      // Send email
      mail_sender($to,$support_mail,$subject,$message) ;
    ?>
Member Avatar for iamthwee

Quit trying to do the whole thing all at once. You gotta backtrack.

First, just try to get a simple text file uploaded to that server with php. Don't even create the file with php use one already made right inside the same directory you're inside.

Try with ssl, and try the port.

Eliminate one thing at a time... It's the only way.

Ok a couple of things that ma help. Curl is designed to get files not send them so drop that approach all together. SInce your not on a windows (yea) then I am assuming you have access to ssh and a command line or as you said you want to run it from cron. Here is a shell script that will transfer a file to another server via ftp:

#!/bin/bash    
# $1 is the file name for the you want to tranfer
# usage: this_script  <filename>
IP_address="xx.xxx.xx.xx"
username="remote_ftp_username"
domain = sample.domain.ftp
password= password

ftp -n > ftp_$$.log <<EOF
 verbose
 open $IP_address
 USER $username $password
 put $1
 bye
EOF

You then use crontab -e to edit the crontab and call the script:

05  00  * * * /my/path/to/file/this-script dafile.csv 2&>1

iamthwee - Thank you for your reply

I already followed that approach, which is why I have gotten to where I am right now. Everything is working exactly the way that it is supposed to work with the exception of getting the file (which is already created) over from my server to the Secure server of the vendor. And the current issue is that I can't seem to make a connection with their server through the script.

The login on line 108 is what fails, and drops through to send me the error email with the Subject Line 'REMOTE LOGIN FAILED....'

So lines 110 - 119 never get a chance to even see if they work.

That is where I'm at with it.

********************************
rch1231 - Thank you for your reply as well

I have already evolved away from cURL for anything other than verifying that the file hasn't already been created.

Just one question : What language is that written in, that you posted?
I don't recognize any of it at all

Maybe I need to be asking in a different thread since this one is titled using cURL as part of the title... that isn't the issue at all any more.

Member Avatar for diafol

Are you allowed to write a script file to the vendor's secure server?

The you could "pull" a file instead of "pushing" it.

You could do file_get_contents() and then a file_put_contents()

The cron job could trigger a call from your server, e.g.

$date = date('Y-m');
$token = ... (some hash)...;
$file = "data_$date.csv";
header("Location: http://www.example.com/showman/pickup.php?f=$file&token=$token");

OR use cURL to call !! :)

The remote file - something like:

$url = $_GET['f'];
//do some token / key authentication and cleaning
$file = file_get_contents($url);
file_put_contents('dir/subdir/filename',$file);

Anyway. That's thinking aloud.

commented: That might be the best suggestion yet. +2

Now that is something that might just work...

Although I do know that cURL can be used to read or write files, I understand that reading them remotely is more common then writing them remotely, so I will have a chat with the product vendor to see what the chances are that they could just have their IR guys write/run a little nightly cron job to fetch my file after I create it...

That should be a really simple solution.

Thanks
Douglas

Will let you know how it all works out.

Member Avatar for diafol

Well, if the vendor guys will do that, then all bets are on - curl, file_*, whatever. I think :)

I should state that this is just an idea - I've never done this from your perspective, but I can't see why it wouldn't work.

Yep, I understand that it was an idea, but one that I hadn't come up with...

But I just came up with another option as well.

I hired a company last year when I hit a brick wall on a payment processing script, and for a couple hundred bucks, he resolved the issue and provided the code and eliminated my headache... LOL

Well, I just picked up the phone and called him, so he is going to see what he can come up with for me. Just need to get it behind me and move on to other pressing matters.

Thanks again for your feedback.
Douglas

That is just a Unix/linux shell script that can be invoked one line at a time from the command line (if needed) for testing. If you notice the first line is #!/bin/bash wheich tells the OS what to use to run the script.

The other thing that I noticed is you keep referencing that the vendor has a secure ftp server and all of us have been referencing the ftp application instead of sftp. Try the shell script like this:

#!/bin/bash
# $1 is the file name for the you want to tranfer
# usage: this_script <filename>
FILE=path/file
IP_ADDRESS="xx.xxx.xx.xx"
USERNAME="remote_ftp_username"
domain = sample.domain.ftp
PASSWORD= password
/usr/bin/expect<<EOD
spawn /usr/bin/sftp $USERNAME@$IP_ADDRESS
expect "assword:"
send "$PASSWORD\r"
expect "sftp>"
send "put $FILE\r"
expect "sftp>"
send "bye\r"
EOD

The reason for expect "assword:" is that in some cases it responds with a capital P and others a lower case p and this will accept either one.

***  SUCCESS   ***

Here was the resolution (I'm only half bald now, so found it in time)

There were actually 2 things that needed to be addressed.
The first was actually getting connected to the remote server, which finally happened because they changed some settings on the server. They wouldn't say what the changed, but it worked.

The second issue became that writing to the file continually failed and I tried every combination of variables I could think of...

Then I searched and found a way to display any errors it was running into using

print_r(error_get_last())

(never used that before)

Turns out I was defining the source file as just the filename, which I had just created.... WRONG

It needed the full path to that file. If that would have been defined in any one of the dozens of references I read through in my searching for the answer, it would have happened more quickly.

    $source_file=$_SERVER['DOCUMENT_ROOT'].$directory.$filename;

That, as the definition of the source file worked perfectly...

Thanks again for all the feedback in here. I love having a great support system like this.

Here is that section of code in total, so it makes sense and maybe will help someone in the future...
Didn't need cURL afterall...

        // the $output variable contains the .csv file I built earlier in the script
        // This creates the .csv file on my server
        // and used for future verification that a daily file has not already been sent.
        $fd = fopen ( $_SERVER['DOCUMENT_ROOT'].$directory.$filename, "w");// open file to write to
        fputs($fd, $output);// write the data to the opened file
        fclose($fd); // Close the file

        //Create connection to vendor secure server
        $source_file=$_SERVER['DOCUMENT_ROOT'].$directory.$filename;
        $dest_file_path = '/'.$filename;
        $ftp_conn = ftp_connect('ftp.vendorftpurl.com',21);
        $user='username';
        $pass='password';
        //Login

        if(ftp_login($ftp_conn, $user, $pass)){
          //Set passive mode
          ftp_pasv( $ftp_conn, true );
          // Transfer file
          $transfer_result = ftp_put( $ftp_conn, $dest_file_path, $source_file, FTP_BINARY );
          $last_error = print_r(error_get_last());
          ftp_close($ftp_conn);
          //Verify if transfer was successfully made
          if( $transfer_result){
            $subject="Successful at ";// successful tranfer of data file
          }else{
            $subject="Failed at ";//tried to send file and failed
          }
        }else{
          $subject="REMOTE LOGIN FAILED when ";//tried to login remotely - failed
        }
    // variable $subject was used in email to myself reporting success/failure

Thanks again... This one is SOLVED...

Sorry rch1231 I don't understand any of that, and don't see where it comes into play with what I was trying to do using a PHP script in a CRON job.

But thanks for responding anyway.

Member Avatar for diafol

Great SM - glad you got it sorted :)

Thanks.. and the best part was I didn't have to pay someone else to fix it for me...

Member Avatar for iamthwee

Thank God you solved it. It was beginning to irriate me...

So in the end it was a simple path issue.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.