Page 1 of 2
Possible nFS Combining and Uncombining Solutions
Posted: Fri Jul 11, 2008 8:52 am
I started this thread to continue the discussion from another thread.
One of the biggest concerns with nFS is the combining and uncombining of records. There are problems with some folders containing to many records and this slows things to a crawl. There are problems with folders where combining errors create folders which have multiple different individuals in them and we have the task of seperating out the different individuals.
What are your ideas on how you feel these processes could be improved so they function more efficiently?
We hear a lot of complaining about the problems, lets discuss your ideas for possible solutions.
Posted: Fri Jul 11, 2008 9:13 am
I'd like to see a block on combining records when it would result in a impossible combination - such as a son who is his own father. Or two individuals who are simply too far apart in time or space to possibly be the same. The error would have to be resolved or bad data disputed before the combining could take place.
At least that keeps a bad problem from getting much worse before it gets sorted out. Also it requires more mouse clicks to mess things up.
Russell's corollary #2 to Murphy's law: If it's possible to mess things up with a singe mouse click, some one will click it.
Posted: Fri Jul 11, 2008 9:43 am
I wonder if something like how programmers use 'deprecated' could be used. If you could mark a record as 'bad' or 'really this other person', then you could lock the bad records without actually merging.
Specifically, if I had 3 J. Smiths, with no other information, It seems silly to 'merge' those records, when all that is different is an IGI # or other identifier. Why not archive / delete / make static 2 of the 3, and mark them appropriately. Marked records would be read only, and maybe not even stored in the same way as active records. They could be flattened to raw text and stored / displayed as a note, or attached file or somesuch.
This tagging would also help with erroneous information, as you could note the reason for the bad record. You might have a record like 'J. Smith, Birthdate incorrect on marriage record, correct birthday in record 384883 from family bible'.
Marking records in this way is MUCH more intrusive than merging. I am not sure if it would be possible to 'unmark' the records easily, especially if they were flattened to text as I recommend above.
That being the case, I would probably allow users to vote for the tag, but only allow moderators to actually flatten the record.
This might be very useful for well know / vetted lines like royalty and church historical figures.
A bit rambling, but there it is:
Possible solutions - my 2 cents
Posted: Fri Jul 11, 2008 10:00 am
There are several example combining problems I've come across, and each has a different solution which comes to mind. These are based on my limited understanding of how the NFS system works, and my background in software engineering.
For some of instances where I'd like to separate individuals, I can see a single field where the difference lies (for example, name of spouse). In these cases, just being able to sort the merged records by a certain field would speed up splitting. This would be equivalent to adding an ORDER BY statement onto a SQL query (or whatever is being used to retrieve the records from storage).
In other instances, the problem is hugely compounded by overzealous combining (fathers combined with sons combined with grandparents and other unrelated people, etc.). These are the ones which seem to be causing the biggest problem. My solution for this last one would be to change from the current "View Combined Records" approach which displays only the last 5 individuals to a tree view of the combine events. Rather than picking off individual records one at a time, what is usually wanted is the ability to find a single combine point in the tree and undo only that one. Thus, a merge which combined a person with 300 records with a (wrong) person of 100 records doesn't require working through a stack of 400 records, pulling out 100 people one at a time. The only problem to that is whether or not the system is currently storing the merge events (so as to be able to construct the tree). Whether or not the data is historically available, it could be added and used from here on.
The third combine error I've come across consist of single records which list their point of origin as Ancestral File, and are clearly containing information about two distinct individuals. These usually cause the system to recommend records from both individuals as being matches (which makes sense). However, there doesn't seem to be any way to unlink the problem record. These are the times when all out destruction of the bad record would be nice.
I also agree with the need for having a block preventing separated individuals from being combined. There is such a block in place for individuals with Church Membership records.
And with all of these, it would be very valuable to have the ability to flag individuals such that I either have a readily available report which shows when any are combined or split by other people, or have a notification (email or NFS internal notice) when such an event occurs.
Posted: Fri Jul 11, 2008 1:23 pm
Another variant on this would be to be able to work on groups of r ecords where you know there are a couple hundred separate entries for that name but you know how many can be in one folder.
I heard there is a knowledge document out there that says the .93 release has the limit at 80 records per folder, so if say there were 200 entries (I have one possibly myself in one of my lines) then there should be a way to have multiple folders to bring together like entries as well, where it is known that everything is very nearly the same but the problem is there are just so darn many entries! And a way to see that you have multiple folders on the individual so you know what you are working on.
The last thing is that when you find temple work on one entry anywhere in any of the folders you have open on an individual should this be a viable workaround for the IOUS problem pending further developments on this, it should also be able to carry over to the multiple folders, so that someone doesn't just come along and say 'there's temple work here', and redo the work. In most cases in combining the duplicates a duplicate set of ordinances will do it but there's no guarantee that will work every last time either.
Posted: Fri Jul 11, 2008 3:41 pm
I like the idea Earl gave of Archiving some of the records. I think that once a folder contains completed ordinances and someone has selected the correct entries in the Summary area. All other records which are not used should be archived and not loaded each time the folder is loaded. The archived records could be loaded if someone clicks on the Details tab or the combined records view, but it would sure make things run eaiser if only the summary information and ordinance records were loaded unless someone requests to see the other combined records. Then when we are just moving through the pedigrees we wouldn't have to wait while all the duplicates are loaded for each folder.
I know the Church will not want to allow users to delete or change someone elses submissions but if we could also identify bad records and mark them somehow it would help. (until the owner claims them and corrects them) Then those bad records would not be used in decisions to merge.
Some other options might be:
Archive unclaimed records until someone claims them
Allow users to mark records to be archived
Allow several users working together to mark a record as bad and archive it
Posted: Fri Jul 11, 2008 4:01 pm
Like Rick I would like to have the ability to uncombine blocks of similar records. If the list of duplicates were displayed in spreedsheet format with the headings at the top instead of along the side and we had the ability to click on a heading and have all the records be sorted by an event, like a birth date. Then all the records with the incorrect birth date would be together in a block, then if they would allow us to select a block of records to uncombine and put them all into one folder then we wouldn't have to go through the combining process to get them all back in the right folders.
Example: I have an ancestors folder with over 900 names in it. There are at least 3 different individuals (father, son, and grandson) that I have identified in that folder with 100's of records for each of the 3 individuals. If we could sort by birthdate then mark a block of records to be split off into a new folder if would be so much better than separating them one at a time then combining them one at a time back into the correct folder.
Posted: Fri Jul 11, 2008 4:31 pm
When I originally brought this issue up in a different thread I had in mind the idea of separating blocks of records. Most often I see two or three distinct individuals (and their respective combined records) being combined as one individual. To sort them out into the two or three individuals as a block of records would be ideal. I also liked Rick's idea of being able to locate the point at which the mistaken combine point was made and just undoing that action. These things could save us all a lot of time. Time that I feel would be better spent on researching and finding new individuals and going to the temple rather than cleaning up mistakes in the NFS database.
Combining records and separating out errors.
Posted: Fri Jul 11, 2008 4:43 pm
I agree with some of Rick Parker's comments. I spent more than a week trying to make sense out of my PA Quaker Moores and Millers which were all scrambled because of loops within loops. Before the 80 record limitation, I separated records from 5 different generations until I identified the problem. Someone identified as bbull2810432 had submitted information to Pedigree Resource File which contained multiple parents. He had one person connected to a spouse 3 or 4 generations away, and another married to his grandmother as well as his wife, as well as others linking mistakes. I know of al least 5 linking errors in his submission, but feedback resulted only in my being told that we have to wait until that person gets access to correct his own errors. I separated out about 5 generations of his submission in order to get the family to appear correctly in nFS...but someone has gone back in and recombined some of the names again. I wish there were a way to quarantine a specific record that is known to have erroneous links within it, if it can't be deleted. When I consider how many hours I spent working on this problem, I think the developers should consider our worthy efforts to create a correct database, and stop protecting those who have made a mess of it.
Posted: Thu Jul 17, 2008 5:46 pm
GarysTurn wrote:I think that once a folder contains completed ordinances and someone has selected the correct entries in the Summary area. All other records which are not used should be archived and not loaded each time the folder is loaded.
I need to submit this formally through the nFS feedback system, but there also needs to be a way to select which marriage information to display.
Additionally, when there is more than one ordinance date for a given ordinance, we need either the ability to select which date to display (a la "Summary"), or the system should automatically display the earliest date for a given ordinance.
I have an example in my line of proxy initiatory occurring several years AFTER proxy endowment which is itself several decades AFTER live sealing to spouse (for some reason the "live" endowment was never recorded on the membership record, or the temple records in question have not been extracted that far back or are otherwise not correctly linked to this individual).