943,192 Members | Top Members by Rank

Ad:
  • C Discussion Thread
  • Marked Solved
  • Views: 5353
  • C RSS
Mar 21st, 2010
0

LibCurl - Copy webpage contents to String

Expand Post »
Hi There.

I am trying to use libCurl to copy the contents of a webpage into a string in C.

This is what I have so far:

  1. void Cmd_translate_f (char *message)
  2. {
  3. CURL *curl;
  4. CURLcode res;
  5.  
  6. curl = curl_easy_init();
  7. if(curl) {
  8. curl_easy_setopt(curl, CURLOPT_URL, va("http://ajax.googleapis.com/ajax/services/language/translate?v=1.0&q=%s!&langpair=en|fr"),message);
  9. curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
  10. res = curl_easy_perform(curl);
  11. }
  12.  
  13. }

The function takes a string and then uses the google translating service to translate it. I am, however, unaware of how to copy the contents of the received webpage into a c string. All this code does at the moment is print out the contents.

Many Thanks :)
Similar Threads
Reputation Points: 10
Solved Threads: 0
Newbie Poster
icewolf is offline Offline
4 posts
since Mar 2010
Apr 2nd, 2010
1

how to read lines of the web content?

i have the same question. technically i want to fetch .csv files from an url with libcurl. yes it works: after compiling the terminal print the .csv content.

however i have no clue how to get the "res" variable (CURLcode) into a string that might be treated further.

it would be perfect if the "res" thing could read by ifstream or something (to use getline or something). at the end of the day, i want to put parts of the .csv web content into a sql query string.
Reputation Points: 9
Solved Threads: 1
Newbie Poster
nigecus is offline Offline
2 posts
since Apr 2010
Apr 3rd, 2010
1
Re: LibCurl - Copy webpage contents to String
I've never used libcurl. I'd love to help you, but all i can do is read the libcurl example that does exactly what you're looking for, and repeat what it says, but that would be kind of dumb and a waste of everyone's time, since you can just go read it yourself.
Last edited by jephthah; Apr 3rd, 2010 at 2:31 am.
Reputation Points: 2143
Solved Threads: 178
Posting Maven
jephthah is offline Offline
2,567 posts
since Feb 2008
Apr 3rd, 2010
-1

it works now

try this:

  1.  
  2. /* g++ -c -I/usr/include/curl testcurl.c
  3.  * g++ -o testtest testcurl.o -L/usr/lib/libcurl -lcurl
  4.  * ./testtest
  5. */
  6.  
  7. #include <fstream>
  8. #include <vector>
  9. #include <iostream>
  10. #include <sstream>
  11. #include <string>
  12. #include <stdio.h>
  13. #include "/usr/include/curl/curl.h"
  14. #include "/usr/include/curl/easy.h"
  15.  
  16. using namespace std;
  17.  
  18.  
  19. int writer(char *data, size_t size, size_t nmemb, string *buffer){
  20. fprintf(stderr,"Hello I am a function pointer\n");
  21. int result = 0;
  22. if(buffer != NULL) {
  23. buffer -> append(data, size * nmemb);
  24. result = size * nmemb;
  25. }
  26. return result;
  27. }
  28.  
  29.  
  30.  
  31. int main ()
  32. {
  33. /* (A) Variable Declaration */
  34. CURL *curl; /* That is the connection, see: curl_easy_init */
  35. /*CURLcode res; /* Not needed, see: curl_easy_cleanup */
  36. string buffer; /* See: CURLOPT_WRITEDATA */
  37.  
  38. /* (B) Initilise web query */
  39. curl = curl_easy_init();
  40.  
  41. /* (C) Set Options of the web query
  42. * See also: http://curl.haxx.se/libcurl/c/curl_easy_setopt.html */
  43. if (curl){
  44. curl_easy_setopt(curl, CURLOPT_URL, "http://ichart.finance.yahoo.com/table.csv?s=DAI.DE&a=NaN&b=02&c=pr-2&g=d&ignore=.csv");
  45. curl_easy_setopt(curl, CURLOPT_HEADER, 0); /* No we don't need the Header of the web content. Set to 0 and curl ignores the first line */
  46. curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 0); /* Don't follow anything else than the particular url requested*/
  47. curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writer); /* Function Pointer "writer" manages the required buffer size */
  48. curl_easy_setopt(curl, CURLOPT_WRITEDATA, &buffer ); /* Data Pointer &buffer stores downloaded web content */
  49. }
  50. else{
  51. fprintf(stderr,"Error 1\n"); /* Badly written error message */
  52. return 0;
  53. }
  54.  
  55. /* (D) Fetch the data */
  56. curl_easy_perform(curl);
  57. /* res = curl_easy_perform(curl); */
  58. /* There is no need for "res" as it is just a flag */
  59.  
  60. /* (E) Close the connection */
  61. curl_easy_cleanup(curl);
  62.  
  63. /* (F) Transform &buffer into a istringstream object */
  64. std::istringstream iss(buffer);
  65.  
  66. string line, item;
  67. int linenum = 0;
  68. while (getline (iss, line)){
  69. linenum++; /* Move to Next Line */
  70. cout << "\nLine #" << linenum << ":" << endl; /* Terminal Printout */
  71. std::istringstream linestream(line); /* Read Next Line */
  72.  
  73. int itemnum = 0;
  74. while (getline (linestream, item, ',')){
  75. itemnum++;
  76. cout << "Item #" << itemnum << ": " << item << endl; /* Terminal Printout */
  77. } // End WHILE (items)
  78. } //End WHILE (lines)
  79.  
  80. return 0;
  81. }
Reputation Points: 9
Solved Threads: 1
Newbie Poster
nigecus is offline Offline
2 posts
since Apr 2010
Apr 4th, 2010
0
Re: LibCurl - Copy webpage contents to String
Eventually got it working.
  1. size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
  2. size_t written;
  3. written = fwrite(ptr, size, nmemb, stream);
  4. return written;
  5. }
  6.  
  7. void FuncBlahBlah {
  8. curl = curl_easy_init();
  9. if(curl) {
  10. fp = fopen("C:\\trans.txt","wb");
  11. curl_easy_setopt(curl, CURLOPT_URL, "http://ajax.googleapis.com/ajax/services/language/translate?v=1.0&q=hello&langpair=en|fr");
  12. curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
  13. curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
  14. res = curl_easy_perform(curl);
  15. curl_easy_cleanup(curl);
  16. fclose(fp);
  17. }
  18. fp = fopen("C:\\trans.txt","rb");
  19. fseek (fp , 0 , SEEK_END);
  20. lSize = ftell (fp);
  21. rewind (fp);
  22. fread (translated,1,lSize,fp);
  23. translated[lSize] = 0;
  24. fclose(fp);
  25. }

The only way i could get it to work was to save the output to a file and re-read it and put it in a string. It does work great though. Hope this helps.
Last edited by icewolf; Apr 4th, 2010 at 4:35 pm.
Reputation Points: 10
Solved Threads: 0
Newbie Poster
icewolf is offline Offline
4 posts
since Mar 2010

This thread is solved

Either the thread starter or a moderator has marked this thread as solved. You can most likely trust the responses and answers given. There is most likely no reason for any further responses to be posted here. If you have a related question, please start a new thread in this forum instead.

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.
Message:
Previous Thread in C Forum Timeline: How do you make a function return a string value?
Next Thread in C Forum Timeline: Strange problem with for loop and labs





About Us | Contact Us | Advertise | Acceptable Use Policy
Forum Index | Build Custom RSS Feed


Follow us on Twitter


© 2011 DaniWeb® LLC