I have a csv script that runs within a sequence on a set of gathered urls like so: threaded(urls, write_csv, num_threads=5). The script writes to the csv but seems to rewrite the first row for each url rather than writing to new rows for each subsequent url. I'm new to python and Dani; Do I need to add a counter and index to accomplish this or restructure the program entirely? Here's the relevant code and thanks in advance:

import csv
from thready import threaded

def get_links():
    #gather urls
    threaded(urls, write_csv, num_threads=5)

def write_csv(url):
    data = {
            'scrapeUrl': url,
            'model': final_model_num,
            'title': final_name, 
            'description': final_description, 
            'price': str(final_price), 
            'image': final_first_image, 
            'additional_image': final_images,
            'quantity': '1', 
            'subtract': '1', 
            'minimum': '1', 
            'status': '1', 
            'shipping': '1' 
        with open("local/file1.csv", "w") as f:
            writer=csv.writer(f, delimiter=",")
            writer.writerow([s.encode('ascii', 'ignore') for s in data.values()])

if __name__ == '__main__':

Edited by Joe_6

4 Years
Discussion Span
Last Post by Joe_6

You pass in write_csv an url, but I do not understand where those other variables like final_model_num are coming.


I didn't include the entire script just the parts that are relevant in an effort to be concise; all variables are properly defined and the script works otherwise as intended except for the fact that the data written to the csv file is one line that represents the desired output for the last url rather than many lines for each url being passed. I can include the entire script upon request.

This question has already been answered. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.