Mysql Duplicate Key Issue With Product Importer

[Binarydog]

Hi all, I'm hoping someone here might be able to give me some advice. My current client has hired me to create a product importer for their shopping cart. The problem I am having is that there is a total of 7,154 products which need to be inserted into the database, but so far I can only get just over 1,000 of them to insert successfully.

Because I'm working on a test version of the site I can clear out the database and start again. But no matter how many times I do this I still hit the same problem which is after the first 1,000 or so products have been inserted Mysql generates a Duplicate Key Entry error and then the processing stops.

I have tried using INSERT IGNORE which at least lets the php script continue executing, but the final amount of products inserted still remains the same. Has anyone else had an issue like this and managed to resolve it? It's got me stumped.

[pritaeas]

Did your client not use the same primary key? Ignoring duplicates will not add them, if they are found. I think your ebst bet would be to remove your key in your test database, and then find out which records can be removed. Without the key, all records should be inserted without a problem.

[veedeoo]

Hi,

This problem can be associated with your php time out limit.. there should be no reason for duplicates unless your php timed out and lost its communication with the mysql server(shared hosting only). Otherwise, you can try LOAD DATA INFILE function as described here. This can take a text file.

Double check your php.ini file settings for max_execution_time and max_input_time values. Try increasing these values above your default settings. An ideal settings for sites that are accepting a minimum of 100MB video uploads is set to at least 1000 for both. So, in your case, this can be a reasonable settings.

Another one is a fucntion called set_time_limit(), but this one requires the safe mode to be off.

[Binarydog]

Thanks for the replies guys.

@pritaeas. The database table in question has an auto incrementing primary key and I dont see why I should have to change this.

@ veedeoo. My client has his own dedicated server that I can and have set the time limit to zero so the script can work without timing out. I haven't tried the LOAD DATA INFILE function though, so I will give that a try next.