Using Web Tools to Map the Members

Use this forum to discuss issues that are not found in any of the other clerk and stake technology specialist forums.
RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Thu Jun 19, 2008 10:57 am

The Earl wrote:The geocode field in MLS is explicity for local use, and does not sound like it will ever contain lat/long. If I were grouping people in MLS, I would explicity put something not lat/long in the geocode field, as lat/long groups are only useful with boundaries, and are not easy to interpret.

I would also not worry about obsolecence. The 'hints' that I have seen do not imply a tool similar to the one you are making, nor are they likely to come out soon.

I am almost sorry to hear that. Reading between the lines, I had been thinking and half-expecting that church IT would add new lat/lon fields to MLS and provide a means of geodcoding addresses to populate those fields.

User avatar
mkmurray
Senior Member
Posts: 3241
Joined: Tue Jan 23, 2007 9:56 pm
Location: Utah
Contact:

Postby mkmurray » Thu Jun 19, 2008 12:58 pm

boomerbubba wrote:I am almost sorry to hear that. Reading between the lines, I had been thinking and half-expecting that church IT would add new lat/lon fields to MLS and provide a means of geodcoding addresses to populate those fields.
I kind of think you are both mistaken. My impression was that Church IT is putting something together for the LUWS, not MLS. It didn't appear to me that the project was dead either.

I got these impressions from this thread: http://tech.lds.org/forum/showthread.php?t=141

DISCLAIMER: I'm only a volunteer modeartor, not an employee of the Church. I do not have direct knowledge of what projects are currently being worked on at the Church, nor their scheduled or tenative release dates.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Thu Jun 19, 2008 1:49 pm

mkmurray wrote:I kind of think you are both mistaken. My impression was that Church IT is putting something together for the LUWS, not MLS. It didn't appear to me that the project was dead either.

The inherent problem with that speculation is, I think, is this: If all a LUWS enhancement does is render a map of ward members on the fly from the addresses themselves, a significant number of those mapped lat/lon coordinates would be wrong. That might be okay for amateur member-mappers toying with batch-geocoding scripts, but I doubt the church would publish something that dubious on its web site.

The underlying data, which resides in MLS and its parent databases in Salt Lake, has to be scrubbed first to get the error rate down to something acceptable. It makes much more sense to scrub the addresses upstream in MLS or its parent database in Salt Lake, geocode them and capture the lat/lon coordinates there.

Then the church would possess a resource to enable much smarter geographic management for stake/ward reorganizations, etc. Mapping members on the LUWS would just be icing on the cake.

The_Earl
Member
Posts: 278
Joined: Wed Mar 21, 2007 8:12 am

Postby The_Earl » Thu Jun 19, 2008 2:04 pm

mkmurray wrote:I kind of think you are both mistaken. My impression was that Church IT is putting something together for the LUWS, not MLS. It didn't appear to me that the project was dead either.

I got these impressions from this thread: http://tech.lds.org/forum/showthread.php?t=141
I agree entirely with this.

This tool is intended for use stand-alone at the local unit level. I do not think that a LUWS map would serve the same needs as this tool.

That is not to say that whatever the church is working on will not make this tool easier to use, or export a KML file directly, but I think those are secondary or tertiary features that are months away from implementation.

I do not think the church is working on a Emergency Prep mapping application, or a ward boundary move application, or whatnot.
mkmurray wrote: DISCLAIMER: I'm only a volunteer modeartor, not an employee of the Church. I do not have direct knowledge of what projects are currently being worked on at the Church, nor their scheduled or tenative release dates.
Ditto!

User avatar
mkmurray
Senior Member
Posts: 3241
Joined: Tue Jan 23, 2007 9:56 pm
Location: Utah
Contact:

Postby mkmurray » Thu Jun 19, 2008 2:13 pm

Yes, I agree with both of your comments. Thank you.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Mon Jun 23, 2008 7:11 am

I note that Joel Dehlin confirmed a couple of days ago that a member-mapping project is underway. In a post promoting the beta site for the great new Meetinghouse Locator page, he said:
We've created a mapping service which takes the lat/long data for meetinghouses and plots it on Google's or Microsoft's mapping service. In the near future we'll also plot ward membership information.
See http://tech.lds.org/forum/showpost.php? ... tcount=125

I am sure that the church developers are well aware of the GIGO problems that arise in the address data harvested from MLS. I hope they are working on back-end solutions that grapple with this problem, and that the task of scrubbing the data upstream is on the table. (Requiring ward clerks to use country-specific postal standards would help, enabled by address-validation built into MLS. I think ward clerks also could be employed to resolve the warnings and ambiguities that occur with a small percentage of geocoded records. For rural units especially, where geocoding often fails, it would even be nice for the ward clerks to be able to enter GPS-generated coordinates for exceptions.)

I also hope the church will use multiple geocoders to help fill in gaps. An address that one commercial service fails to geocode may be successfully geocoded by another. Unlike amateurs like me, who are trying to do this for free, an institution like the church can feasibly employ such commercially licensed solutions. (Even so-called "rooftop-level" geocoding is available in many developed areas such as the United States.) Apparently some of that is already happening with the new Meetinghouse Locator app.

The resulting lat/lon coordinates should be captured and propagated to new fields in MLS, so the units can use them for their own purposes. Functions such as HT/VT management, emergency planning, and maintenence of fast-offering routes are examples of local unit needs, which likely would not be met by a generalized member-mapping function in LUWS. And of course, once lat/lon fields are captured in MLS, stakes would have a hugely valuable resource for boundary management.

Finally, if there is a mapping function in LUWS for authenticated members to use, I hope it would include ward boundaries, and that both the boundaries and member layers of the map could be downloaded in common formats such as KML, just as the membership directory can be downloaded today.

Just my own modest wish list.

User avatar
Mikerowaved
Community Moderators
Posts: 4262
Joined: Sun Dec 23, 2007 12:56 am
Location: Layton, UT

Postby Mikerowaved » Mon Jun 23, 2008 12:00 pm

Good post boomerbubba, thanks.

I might also add that sometime early in the development process, a common Lat/Lon "standard" should be set for all coordinates used throughout the Church. Whether it be MLS, LUWS, maps.lds.org, or any other project. I am disturbed by the lack of standards among the various GPS units being marketed today and hope the Church does not fall into the same pit. For example, many handheld devices use DM (see below) that must be converted to either DMS or DD to be used with popular geocoding software.

These are the three common formats of displaying Lat/Lon coordinates:

  • DMS Degree:Minute:Second (49°30'02"N, 123°30'30"W) or (49d30m02.5s,-123d30m30.17s)
  • DM Degree:Minute (49°30.0'-123°30.0'), (49d30.0m,-123°30.0')
  • DD Decimal Degree (49.5000°,-123.5000°)
Even within each format are many different styles of displaying the same information. I've only shown a couple. My point is, if CHQ has not already adopted a standard for displaying and storing GPS coordinates, now would be a good time, not after different projects get started that use differing systems.

Mike
So we can better help you, please edit your Profile to include your general location.

russellhltn
Community Administrator
Posts: 30913
Joined: Sat Jan 20, 2007 2:53 pm
Location: U.S.

Postby russellhltn » Tue Jun 24, 2008 1:26 am

Mikerowaved wrote:I might also add that sometime early in the development process, a common Lat/Lon "standard" should be set for all coordinates used throughout the Church. Whether it be MLS, LUWS, maps.lds.org, or any other project. I am disturbed by the lack of standards among the various GPS units being marketed today and hope the Church does not fall into the same pit.
Well, I'd think the church would adopt WGS 84. :D

What you're talking about is the display format. Most GPS units I know of will allow you to set the display format to suit your preferences. I'd like to see the software accept the coordinates in any format and display it in the chosen one. Much like the way PAF will re-format a date into the standard format.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Tue Jun 24, 2008 6:26 am

One last gotcha to keep in mind:

When plotting multiple ward members there will be a significant number that geocode to precisely the same point -- the most salient example being apartment complexes. Street-address geocoding, even some geocoding purported to be "rooftop level," generally knows nothing about apartments. Most address-based geocoding stops at the curb, perhaps with an arbitrary setback from the street..

So there will be multiple dots (or similar markers) legitimately occupying the same point, no matter how closely the user zooms. Some GIS standalone systems and web-based mapping services don't deal with this situation very well. The last point plotted tends to obliterate all the others occupying the same spot on the rendered map display.

I think that is what the Yahoo mapping API does. I don't know how the Google Maps and MS Virtual Earth APIs handle this. I like the way Google Earth behaves: When the user mouses over such a multiply geocoded point, the cursor changes form. A click expands it temporarily into a cluster of individual icons.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

My Solution to Geocoding and Mapping: GeoDB v 1.0 Released Here

Postby RossEvans » Fri Jun 27, 2008 9:50 pm

This post, replicated in a ReadMe file, introduces a system of two related script files - GeocodeFromPalmFamily.vbs and MakeKML.vbs.

The overall objective is to read files extracted from the church MLS system, geocode families for a ward or other unit, and produce several KML output files rendering Folders and Placemarks for loading into Google Earth. The basic file shows all families in a given ward or unit. Optional files also show home-teaching and visiting teaching companionships and assignments.

These KML files, like their underlying source data, are intended for church use only. They should be protected from unauthorized use, and not stored on third-party servers. The content of the basic families map contains no substantive data that is not already downloadable by authorized ward members from the Local Unit Web Sites on LDS.org, so in my own opinion it is suitable for dissemination to ward members. The home- and visiting-teaching files are intended as a tool for leaders in the bishopric, quorums and Relief Society, as well as clerks. If in doubt, follow the guidance of your bishop or other priesthood leader.

The functionality is split into two scripts for good reason: It is better to separate the geocoding step from the KML-rendering step because geocoding is not a simple process that can just be run blind, and the address data is imperfect. Almost certainly there will be ambiguities, errors and warnings to deal with between the initial geocoding process and the use of those lat/lon coordinates to produce accurate maps. Garbage In, Garbage Out. Most often, address errors are best dealt with by correcting them upstream in MLS, then exporting the data again.

In addition, very likely there will be exception addresses that fail to be geocoded satisfactorily by the primary geocoding provider, but which might be geocoded by another provider.

In this case, the first script -- GeocodeFromPalmFamily.vbs -- uses the Yahoo mapping API as the primary source of geocoding. For secondary sources, the user is free to try any of several free sources online. I find it best to use Google Earth itself to try geocoding addresses that Yahoo could not find. (I could not script use of Google Maps' API automatically, because its terms of use preclude this.)
This editing functionality differentiates this system from some batch-geocoding scripts that ignore warnings and geocode some records inaccurately (such as at a zip-code centroid), and exclude other records without warning.

Overall, the process works like this:

1) Extract the file(s) from MLS. This requires administrator access to the MLS system, and obviously should be authorized by the bishop. To produce only a simple map of all families in the unit, only one extract file is required: PalmFamily.csv. To produce the optional files for home and visiting teaching, three other files are required: HomeTeaching.csv, VisitingTeaching.csv, and Membership.csv. All those files must be extracted at the same time.

2) Store the extract file(s) in a dedicated working folder, which will also contain its output KML files and several intermediate files and temp files in csv format, all with column headers and quote-marks to delimit strings. Logically, all these .csv files comprise a relational "database," and really ought to be stored in a proper SQL back end. Since I have not provided one, the scripts juggle the .csv files directly. Yuck.

3) Run the GeocodeFromPalmFamily.vbs script, obviously with Internet access to hit Yahoo.
The script produces a fresh copy of main output file called GeoDB_Households.csv. This file should not be edited, but may be carefully browsed in a spreadsheet such as Excel. (In a future version I might make it read-only.) In addition to the geocoding results from Yahoo -- lat/lon coordinates and standardized addresses -- the script captures three fields that describe the success, failure and uncertainty of each geocoding transaction.

4. The script also produces three intermediate files: GeoDB_GeocodeExceptions.csv, GeoDB_GeocodeWarnings.csv and GeoDB_MissingAddresses.csv. Often the best solution is to correct the addresses upstream in MLS, then export again.

4(a) The exceptions file contains records which Yahoo failed to geocode, or which failed to meet the quality threshhold I have set, despite your best efforts to fix the addresses in MLS. Included in this file are addresses where the street address or street could not be located at all, or which Yahoo only geocoded as the centroid of a zip code, city or state.

I strongly recommend that the user stop and try geocoding these exceptions interactively from a secondary source such as Google Earth, paste the lat/lon into the appropriate fields using a spreadsheet, and save the file again in .csv format. The script is designed to preserve and maintain these exceptions for future use in this database file, so you should not have to repeat this work again for the same addresses the next time you do an update.

4(b) The GeoDB_MissingAddresses.csv file is a report of records that have missing data in one or more of three principal fields -- street address, city or state. (Sometimes the content is present but is in the wrong fields, in which case Yahoo might recognize it anyway and geocode it.) In most cases, of course, these egregious errors are not geocoded at all.

4(c) The GeoDB_GeocodeWarnings.csv is a report of records that have other problems in the address fields that Yahoo had to guess at -- typically misspelled streets, missing directional indicators (N, S, E, W), etc. In my testing, I found that Yahoo usually guesses right. So unless you take steps to the contrary, the scripts will map these addresses as Yahoo suggests.

But sometimes, Yahoo guesses wrong. and substitutes the wrong address . So the addresses get geocoded at the wrong coordinates. Unless we fix such errros upstream in MLS, they will occur and recur. So the right solution is to fix the errors upstream, which should happen anyway because MLS should not contain erroneous addresses, geocoding or no geocoding. Clerks need to get the data right, and this process is a useful tool for finding potential errors in member addresses.

5. After dealing with errors and exceptions, the user is ready to produce the KML files. Run MakeKML.vbs.

The output files will be named according to the unit name entered interactively by the user. There will be a general <UnitName>Families.KML. If the requisite input .csv files exported from MLS are present in the database folder, there also will be a <UnitName>HomeTeaching.KML file and a <UnitName>VisitingTeaching.KML file. All these folders are date-stamped internally according to the date the underlying data extracted from MLS, and these dates are visible within the Google Earth folders.

5(a) At runtime of MakeKMLvbs script, the user has the option to include on the map those records that could not be geocoded, by choosing a default location for these ungeocoded orphans. This is especially helpful in the home- and visiting teaching files, where it may not be useful to leave an assigned family off the map just because it has a bad address. (Or the user may choose to leave them off the map entirely.)

The user is encoouraged to select a meaningful default site such as their meetinghouse or a local landmark, and store its coordinates in GeoDB_DefaultLocation.csv. If the user has not stored such a location but would like to map the orphans anyway, the script will estimate and generate a default site slightly apart from the cluster of geocoded members.

6. The icons for placemarks in the KML files are color-coded: Families in the general file are green. In the home-teaching file, High Priests and their assigned famlies are green, Elders and their families are blue, and unassigned families are orange. Visiting teachers and their assigned sisters are pink, and unassigned sisters are orange. The shapes of the icons also varies: Families and sisters assigned for teaching are pushpins, home teachers are man-shaped figures and visiting teachers are woman-shaped. If a family or sister is mapped at the default location, that icon is an inverted teardrop "paddle" shape.

Within Google Earth, the placemarks also can be navigated within folders displayed in the left pane. The home- and visiting teaching folders are organized as hierarchies -- quorum or auxiliary, then district, then companionship, then individuals. They are often best browsed by checking/unchecking folders to allow companionships to be isolated for display. It is even possible for the user to cut-and-paste interactively from one folder to another in what-if fashion to adjust companionships by geography.

7. Lather, rinse, repeat as your ward roster changes. In my huge ward, with many apartment-dwellers and a lot of turnover, I plan weekly updates. YMMV.

Sorry this is so complicated. It is complicated world. I don't think anyone will be happy for long, once the novelty wears off, with batch-geocoding solutions that don't confront the problems of getting the geocoding right upstream and maintaining this data over time. Producing maps on-the-fly directly from address data just does not work as well.

The scripted user interface is pretty crude -- serialized prompt-response dialog popups. I know that all this complexity really belongs in some application, with a decent GUI and a bulletproof SQL back end. Maybe someday I'll write one. But maybe the church developers will render that unnecessary. Ideally, this geocoding functionality will be integrated into MLS someday soon, driven by commercial-grade geocoding and maintained by ward clerks within the MLS database, and the officially correct coordinates will be available for export.

Meanwhile, here is v 1.0.


Return to “General Clerk Discussions”

Who is online

Users browsing this forum: No registered users and 2 guests