Several thoughts on reasons for duplicate templework:
1. The extraction program is not compared to existing templework. (Does anyone know if there is a plan to change this?)
2. Some records are change after submission to create "standardization". These changes do not take history into account. For example, someone born in Lithuania prior to 1900 shows up as born in the USSR, which did not exist for decades later. The same is true for Germany and other places with new names and/or boundary changes.
3. Submitter records in various US states are also sometimes "corrected" on the IGI to similar town or county names in another state.
4. Spelling variations occur within the same document, or not. Figuring out that Runnells may also go by Runels isn't terribly challenging. However, the same name may also be Reynolds, Randles, Rindel, etc. It takes time for researchers to discover the more far-fetched variations, such as Gregory being spelled Craygor.
5. Researchers differ in what they give more weight based on their experience or inexperience.
6. Records availability changes over time. What might be pulled up today with a few keystrokes in 15 minutes may have been stored in an attic or basement and unavailable to the public until last year.
7. Living families are frequently not particularly close today. (huge, broad brush here!
) People do not tend to know more than one or two generations back and frequently do not know aunts and uncles, let alone great aunts and uncles, etc. This compounds the difficulty of cooperating with unknown potential relatives to research a common ancestor.
8. Some folks use different names at different times. The US standard today is first, middle, last. Some folks also have multiple middle names or use a christening name.
9. Some cultures did not have written records. Relying on verbal tradition creates variation.
10. Family traditions can cause errors. A friend of mine came from a large family. No one in her immediate family was born in the same month--until she bought her siblings' birth certificates! They were all clustered in just a few months. Presumably for financial reasons, the parents selected a birthday for each child that was thereafter celebrated and reported as their birthday for the rest of their lives. Another friend of mine ordered her birth certificate for the first time due to a genealogy class assignment. When she got it, she was shocked to find that nearly 70 years ago she had been adopted. On the other hand, official records can have errors, too.
There are many other reasons for duplication, too.