US Census Cone Search Project Idea

[garysturn]

I have an Idea for a community development project. The idea is to develop a search add-in module for the US Census that could be added to any web site. This project, if approved and developed, could carry the FamilySearch logo. It is a search concept I have named a Cone Search.

I have created a short Slideshow to demonstrate how it would work.

It would need to access the data from the US Census from the FamilySearch Record Search API. Even if the project is not accepted as a Community Development project maybe FamilySearch could use some of the search ideas in some of their other projects.

What do you think?

[thomasjking]

the slide show is good
i think this would be very helpful for genealogy

[jaocon]

Gary,

I like the idea and believe it has a lot of merit as a Community Development project! Along with other posts I appreciate your insightful contributions on this forum. I do have a couple of questions/comments:

1. I would hope the search feature would include name variations?

2. I notice that in the search results you have Father and Mother names; in my experience Census records do not normally contain these (unless of course they are found as child) - where are you proposing this information is pulling from? I believe that this would imply that you could link Census Record A to Census Record B in either the same/previous/next sequential Census?

3. I don't know the status/direction of FamilySearch FamilyTree crosslinking features but would you agree that some Community Development project would be a good place to begin linking US Census records into a family tree structure? I would prefer to be able to crosslink Census records which I have researched to the individuals in my FamilyTree. Since I have already done the research (at least most of the Census research) I am hoping the direction crosslinking is moving will allow this.


I believe that a Community project where user submission of documents/records supporting their tree research would assist in more accurately portraying their rendition of the FamilyTree. I believe the US Census would be a good Data Set from which to start such a project. When I submit/assert that Census Record A line X is the son/daughter of Census Record B line Y, in my rendition of my FamilyTree that source citation should provide some validity.


What I would like to see is some Community Development project that allows me to submit certain family records (not necessarily images but at least an extraction of the text/reference to source that may/may not be crosslinked to some RecordSearch/3rd party image source). Effectively what I hope such a project would do is give a jump to the extraction/indexing effort and at least build a Community 'pre-index' of my family source records which could be double/triple verified/synced when those records get around to actually being extracted/indexed.

4. nFS FamilyTree puzzle question: When is nFS FamilyTree node/individual not a record assertion?

[garysturn]

Gary,

2. I notice that in the search results you have Father and Mother names; in my experience Census records do not normally contain these (unless of course they are found as child) - where are you proposing this information is pulling from? I believe that this would imply that you could link Census Record A to Census Record B in either the same/previous/next sequential Census?

Like you have said, the results would only show parents names when a child is found. The data in my examples is dummy data and does not represent actual families. You would see nothing in the Parents Names fields if a Head of household or Wife were selected. The sample data is bad because I showed both a spouse and parents names, if parents names are available you would not see a spouse. You would also not see a mothers maiden name in the results unless she gave her maiden name as a middle name in the census.

Each census year would need different results details, because the amount of information available is different for each census. The 1850 Census is the first census where spouse and children were included in the census, prior to 1850 only the name of the head of household was taken in the census.

I was not proposing cross census linking.

You would have the ability with this proposal to move from census to census using the same search criteria, just by changing the census selected in the census drop down box. I was going to show that in a slides but forgot.

[garysturn]

1. I would hope the search feature would include name variations?

Besides the ability to see alphabetical name variations in the drop down menus, I would propose including soundex results as well as exact spellings in the search results. (I actually had a soundex button in the sample that toggled on and off but removed it before I made the slideshow to simplify the presentation) I would want this to be an optional search so you would have complete control using the wild cards method below.

I would also propose the use of wild cards in the search.

Examples: ? - for a letter * - Multiple letters

Turner could be found with the following searches.

Turner
T?rner
T*r
Tur*
T?rn*
T??ner

This type of searching could help you find names with extraction errors. I have a grandfather whose name was misspelled when extracted, they extracted an i as and e so if you spell his name correctly in the search his record does not come up.

[garysturn]

3. I don't know the status/direction of FamilySearch FamilyTree crosslinking features but would you agree that some Community Development project would be a good place to begin linking US Census records into a family tree structure? I would prefer to be able to crosslink Census records which I have researched to the individuals in my FamilyTree. Since I have already done the research (at least most of the Census research) I am hoping the direction crosslinking is moving will allow this.

I believe that a Community project where user submission of documents/records supporting their tree research would assist in more accurately portraying their rendition of the FamilyTree. I believe the US Census would be a good Data Set from which to start such a project. When I submit/assert that Census Record A line X is the son/daughter of Census Record B line Y, in my rendition of my FamilyTree that source citation should provide some validity.

What I would like to see is some Community Development project that allows me to submit certain family records (not necessarily images but at least an extraction of the text/reference to source that may/may not be crosslinked to some RecordSearch/3rd party image source). Effectively what I hope such a project would do is give a jump to the extraction/indexing effort and at least build a Community 'pre-index' of my family source records which could be double/triple verified/synced when those records get around to actually being extracted/indexed.

Since it is the plan for some future version of the nFS FamilyTree program to allow linking of original documents to our individuals in FamilyTree, I would not propose that this be included in this project. I agree that this will be a great boost to Family History once this ability is available. The Life Browser over at FamilySearch Labs is an example of such a project.

A wiki type cross linking project linking all census records between different census years is an interesting idea, but not what I am proposing here.

[garysturn]

The power I see in this type of search would be in the ability to view the search results in spreadsheet type lists that could be sorted by multiple headings. I was planning to do some examples but have not found the time to recreate my vision of what I called "Form View". I was planning to include this in the slide show but ran out of time and wanted to post what I had so far. I only mentioned one type of form view in the slides. Examples of different types of searches you could do:

• Obtain a list of all individuals in the census with the same Surname living in the same county or town. (Enter just a Surname, State, County, and City if desired) This could help you construct families.
  
• Obtain a list of all individuals with the same first name in the same county or town. (Enter just a given name, State, County, and City if desired) You often have a females first name, place of birth, year of birth but no maiden name, this might give you clues to find that persons family.
  
• Obtain a list of all individuals from a small town. (Enter State, County, City)

The results of these custom searches would then be loaded into a spreadsheet type form. These forms would be customizable, you could move columns and sort by one or multiple columns to assist in finding the results you are looking for. Heritage Quest has some similar results forms with the ability to sort the data (they only allow sorting by one column).

I am certain that somewhere in the extracted data are included page numbers and line numbers. I would like for this information to be available for sorts. Then we could sort the data by families and could even recreate electronic versions of the actual census forms that could be compared to the original images. I would love to be able to download all the census data for a small town and page through the census pages in readable pages generated from the extracted pages. These could be compared to the original images to find additional info not extracted, or to look for extraction errors.

I would also like the ability to create a PDF version of these search results that would be searchable using the PDF search features. I would also like to see the ability to download search results into an Excel spreadsheet so you could use the Search features in Excel to find the data you are seeking.

[jdlessley]

As I went through the slide show one aspect of the search process I was expecting to see was an audit trail and a way of re-entering the search at any point of that trail to take a different branch. This would help searches in which guesses are being made using the data presented. When the path of the search ends and the results are not as expected then the search can be continued at a point back in the search along another path.

Perhaps that type of functionality could be explored once the project begins. I have other impressions of the project that would best be made once you decide to continue the project. I am only looking at the project, which is a good idea to me, through the perspective of a potential user and how I would expect to be able to use this tool.

[garysturn]

As I went through the slide show one aspect of the search process I was expecting to see was an audit trail and a way of re-entering the search at any point of that trail to take a different branch. This would help searches in which guesses are being made using the data presented. When the path of the search ends and the results are not as expected then the search can be continued at a point back in the search along another path.

Perhaps that type of functionality could be explored once the project begins. I have other impressions of the project that would best be made once you decide to continue the project. I am only looking at the project, which is a good idea to me, through the perspective of a potential user and how I would expect to be able to use this tool.

The way I intend for this to work is that any one of the search items can be changed at any time and the new results would be displayed. The purpose of the #Persons column is to help you decide when you have narrowed the list enough to look at the results list.

The data in the drop down boxes also change based on the search. Example: the Given Names in the list are only given names of people with the Surname entered. If there were no Surname then that drop down list would contain all given names for what ever other search criteria was entered. Example: if just a State and City is entered then the given name drop down list would list all given names in that city if there is no Surname entered.

You do not need to start a search from the top down. You could select a Census year then select just a State and see the total #Persons in that state. Then you could add a Birth year and see a total #Persons born in that year and state. Then you could remove the state and see all the #persons in the entire census born in that year. Almost any combination of searches would be possible. You could select "Form View" at any time and load all the results from the search into a spreadsheet type form and sort it to help find the person or family you are looking for. There would need to be limits on the size of list that could be loaded.

I added a few slides to the slide show that show different possibilities for searching.

[garysturn]

The actual search internally would be done from top to bottem and the #Persons would always show the smallest # of results at the bottom, but the order in which you enter the data does not need to start at the top and go down.

I also thought about the possiblility of allowing the user to rearrange the search fields so the results would be listed in a different order. I think that would be something that could be added after everything was working.

Example;

Surname
Given Name
Birth Year
Birth Place
State Residence
County Residence
City Residence

The above is the way it is shown in the form now. The first result is the number of Surnames in the census then as the search moves down the list the number of hits gets smaller as more criteria is added to the search.


State Residence
County Residence
City Residence
Surname
Given Name
Birth Year
Birth Place

If the search order were changed to the above you would first see the total number of people in the State then the # persons would diminish as you moved down the list.

You do not need to enter something in every field at first, but when you do enter something the results are retabulated from the top down. That is why I called it a Cone Search because the results look like a cone.