So I must build an app that has to do with mail address management and more.
Let's say the user has an excel file with 2 millions of rows (email addresses). I made it the oledb way and the first mistake I made was putting ~500k rows in a datagridview, bad bad mistake. The tiny app turned out to occupy ~700mb of ram.
I ditched the datagridview for now (I will later implement it with virtualization + on demand pages). Now with only the dataset it goes to about 170mb then settles at around 100mb of ram.
I would really appreciate some advice on what's the best way to deal with this kind of files(excel, text, csv all with about 2 mil rows) keeping in mind that I need to verify each row against a regex expression, delete duplicates and export to excel, csv or text files.