I am trying to bulk-load a list of objects into a one-column (primary key) db. The only reason is to remove duplicates. I can't load the list in memory, because the file size is way greater than my memory size.

I use monetdb's COPY-INTO command, but I don't want it to fail when there is a duplicate. I want it to add everything that is not a duplicate and skip the duplicates.

Is there any way to do that with monetdb? Any other way?