Using Web Tools to Map the Members

Use this forum to discuss issues that are not found in any of the other clerk and stake technology specialist forums.
User avatar
Mikerowaved
Community Moderators
Posts: 4218
Joined: Sun Dec 23, 2007 12:56 am
Location: Layton, UT

Postby Mikerowaved » Sat Jun 28, 2008 12:13 pm

First of all, let me congratulate you on an outstanding job. It worked for me right out of the box with just a few glitches, but far better than I expected. Here's a few things I found...

Platform: Windows Vista Ultimate w/SP1
WSScript: 5.7.0.6000
MLS: 2.8.3_13810

1. After finding all my families in the GeoDB_GeocodeExceptions.csv file, I went back to the MLS Export and under Customize, switched on Include State/Province. This cleared up those errors. There are 12 fields there, so you might want to make a note as to which ones you need checked for proper export.

2. Most of my entries in the GeoDB_GeocodeWarnings.csv file had an extra compass letter that was not needed. For example, 123 N Maple St, where the correct address should be 123 Maple St. (This happens a lot in Utah.) Google tends to choke on these, so I'm glad to see Yahoo Maps is a little more tolerant. In each case, it guessed correctly.

3. I got 2 VT pop-up warnings, but not much info to tell my what the problem was. Later I traced it down to two missing addresses in MLS. (Which I already knew about.)

4. The default color of forest green for the membership pushpins caused them to camouflage right in with the overhead view of trees, shrubs, and lawn. :eek: This was easily changed in Google Earth.

5. With every Geocoding server I've used, there have always been slight errors. Google tends to put the pins near the middle of the street, while Yahoo puts them more on the front lawn or driveway, however in some cases, 1 or 2 houses away from the REAL address. Of course, this is not the fault of your script. Fortunately, Google Earth lets me manually move the pins to the correct location and re-save the KML file with the updated coordinates. This can take some time for an entire Unit and it would be VERY nice if somehow these updated coordinates could be preserved so a person wont have to do it every time a new set of files is created. (Of course, hopefully this will be solved when we can save these coordinates right in MLS.)

6. It would sure be nice if the script could remember the default path for the working folder so it doesn't have to be drilled down to each time. (OK, this is getting picky! ;))

Like I said, I'm very impressed with what you've done. I know dealing with the format (or lack thereof) of the MLS export files is a chore-and-a-half just in itself. Thanks!

Mike
So we can better help you, please edit your Profile to include your general location.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Sat Jun 28, 2008 12:52 pm

Mikerowaved wrote:
4. The default color of forest green for the membership pushpins caused them to camouflage right in with the overhead view of trees, shrubs, and lawn. :eek: This was easily changed in Google Earth.


Thanks for your kind remarks, and for your test report.

On the subject of the green icons blending with the background, you have a good point. That icon color (for all families in the general Families file and for HP home teachers and families in the HomeTeaching file) can be altered by changing a couple of constants near the top of the MakeKML.vbs script.

I think canary yellow should work nicely. Edit the value of the two constants FamilyColor and HPColor to "ff7fffff" to change them to canary yellow.

I do not experience the problem, because I always load my KML files into Google Earth -- and distribute them within our ward -- along with a solid ward-boundary layer that is colored yellow-beige, and semi-transparent. So the green icons work fine against that background.

I did not produce the ward-boundary layer via scripts, but in commercial GIS software (Manifold). It appears that it would be pretty easy, albeit not so precise, for a Google Earth user to create their own boundary polygons directly by tracting and clicking on their ward outlines if they know where the lines should be. The .pdf file from CHQ should be a good source to copy.. The straighter the lines being traced, the easier it would be. (Zoom in and be careful to keep on street centerlines, for example, if those comprise a boundary.)

In any event, I do see the problem you describe for most users. Hope the change above solves it.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Sat Jun 28, 2008 2:16 pm

Mikerowaved wrote:. With every Geocoding server I've used, there have always been slight errors. Google tends to put the pins near the middle of the street, while Yahoo puts them more on the front lawn or driveway, however in some cases, 1 or 2 houses away from the REAL address. Of course, this is not the fault of your script. Fortunately, Google Earth lets me manually move the pins to the correct location and re-save the KML file with the updated coordinates. This can take some time for an entire Unit and it would be VERY nice if somehow these updated coordinates could be preserved so a person wont have to do it every time a new set of files is created. (Of course, hopefully this will be solved when we can save these coordinates right in MLS.)


1) Yes, the underlying problem is inherent in the provider's address-based geocoding, which typically is based on interpolating within a range of addresses. The geocoding engine just knows that an address range for a street segment begins at one point and ends at another.

The industriy's answer to this problem is so-called "rooftop-level geocoding," which folds in the geography of real-estate parcels from county tax records as polygons. (This is really a misnomer, because the software has no knowledge of rooftops or buildings, and usually reports a parcel centroid. But it is pretty close for single-family homes.) Some of the better geocoding companies include this now, although the coverage is still only about half the United States.

I don't have experience with this high-grade commercial geocoding, but I probably will get a taste this week, when I plan to submit our ward's 550+ addresses to a high-end vendor, Proxix. With the bishop's approval, we are going to try this at least once, primarily to get a complete set of CASS-compliant addresses to help us clean up our very messy membership data in MLS. As a side benefit I will get the vendor's parcel-enhanced geocodes, which I can compare to Yahoo's. If the results are a lot better, I might even build this table into my copy of this system as the preferred lat/lon for an address. The tricky part is keeping this current at an affordable price, given turnover. I continue to hope that CHQ is pursuing such solutions centrally.

2) On your wish for saving lat/lon to override the geocodiing from Yahoo, that is doable. In fact, I considered changing the order of precedence to use a lat/lon stored in GeoDB_GeocodeExceptions.csv so that it trumps Yahoo's. Right now the reverse is true, and the exceptions file is treated as a backstop for addresses that were not geocoded by Yahoo. The order of precedence is Yahoo -> Exception -> Default. Or I could create another table (file) expressly for this purpose.

Such functionality might help a lot in rural wards where the quality of geocoding is only fair. These days, with GPS-enabled devices becoming more ubiquitous, it becomes easier for the member, home teacher or clerk to park in the driveway and push a button to capture an exact coordinate. Almost uniquely among modern institutions, the church has the manpower and organization to conduct such a census on the ground.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Sat Jun 28, 2008 3:42 pm

boomerbubba wrote:I think canary yellow should work nicely. Edit the value of the two constants FamilyColor and HPColor to "ff7fffff" to change them to canary yellow.
...
In any event, I do see the problem you describe for most users. Hope the change above solves it.

Oops. I that was too quick and too dirty. All it does is change the color to lime green. I think I also need to change the href value of the icons to a yellow base icon. Which means I really need to do some minor surgery on the code to enable canary-yellow coloring of these icons.

I will plan to do this in the first maintenance release.

Thanks again to Mikerowaved for his critiique.

User avatar
Mikerowaved
Community Moderators
Posts: 4218
Joined: Sun Dec 23, 2007 12:56 am
Location: Layton, UT

Postby Mikerowaved » Sat Jun 28, 2008 3:49 pm

boomerbubba wrote:2) On your wish for saving lat/lon to override the geocodiing from Yahoo, that is doable. In fact, I considered changing the order of precedence to use a lat/lon stored in GeoDB_GeocodeExceptions.csv so that it trumps Yahoo's. Right now the reverse is true, and the exceptions file is treated as a backstop for addresses that were not geocoded by Yahoo. The order of precedence is Yahoo -> Exception -> Default. Or I could create another table (file) expressly for this purpose.

The Google Earth hand "tweaked" pin positions are re-saved back in the Unit_family.kml file, so that might be a good place to set the highest precedence. A quick check to see if the addresses have already been resolved could also speed up the process of pinging the entire addresses list off the geocoding server. If I want a fresh set of geocodes, I can always delete (or rename) the KML files forcing them to be recreated.
So we can better help you, please edit your Profile to include your general location.

danpass
Member
Posts: 491
Joined: Wed Jan 24, 2007 5:38 pm
Location: Oregon City, OR
Contact:

Suggestions

Postby danpass » Sat Jun 28, 2008 4:12 pm

boomerbubba wrote:These KML files, like their underlying source data, are intended for church use only. They should be protected from unauthorized use, and not stored on third-party servers. The content of the basic families map contains no substantive data that is not already downloadable by authorized ward members from the Local Unit Web Sites on LDS.org, so in my own opinion it is suitable for dissemination to ward members. ..... If in doubt, follow the guidance of your bishop or other priesthood leader.
.
.
1) Extract the file(s) from MLS. This requires administrator access to the MLS system, and obviously should be authorized by the bishop. To produce only a simple map of all families in the unit, only one extract file is required: PalmFamily.csv. To produce the optional files for home and visiting teaching, three other files are required: HomeTeaching.csv, VisitingTeaching.csv, and Membership.csv. All those files must be extracted at the same time.


This is great! Thanks for doing this. If you are in agreement with the following, you might consider including something like this in your readme:

Members can optionally ask their ward web site administrator to exclude all or part of their contact information from the LUWS. Depending on the intended purpose and on how wide the intended distribution of the kml files to be produced by your scripts is, there may be reason to make similar exclusions from the MLS data before it is released for use in this process. The same consideration should be given regarding members who have indicated that they do not want contact from the church. I'm not saying that they should always be excluded. For example, if the intended purpose is for mapping emergency preparedness groups and assignments, you would probably want to include them.

Although mentioned in this or other threads in this forum, your readme probably ought to mention that the administrator who extracts the data from MLS should remove confidential data from the csv files, particularly the membership.csv file. I suggest that you indicate which columns are needed by your scripts along with instruction to the MLS admin to delete all other columns before releasing the data.

User avatar
mkmurray
Senior Member
Posts: 3241
Joined: Tue Jan 23, 2007 9:56 pm
Location: Utah
Contact:

Postby mkmurray » Sat Jun 28, 2008 4:44 pm

I think you ought to consider making an open source project geared around your scripts. Perhaps this could evolve into a desktop application that saves all your settings, modifications, preferences, etc.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Sat Jun 28, 2008 7:06 pm

danpass wrote:This is great! Thanks for doing this. If you are in agreement with the following, you might consider including something like this in your readme:

Members can optionally ask their ward web site administrator to exclude all or part of their contact information from the LUWS. Depending on the intended purpose and on how wide the intended distribution of the kml files to be produced by your scripts is, there may be reason to make similar exclusions from the MLS data before it is released for use in this process. The same consideration should be given regarding members who have indicated that they do not want contact from the church. I'm not saying that they should always be excluded. For example, if the intended purpose is for mapping emergency preparedness groups and assignments, you would probably want to include them.
.


I am in agreement with the intent of respecting members' wishes to be excluded, but I am not well educated about the mechanics of that. How precisely are those wishes executed within MLS? Does it only affect LUWS?

My inherent problem is that my application is driven by the data that is exported directly in those MLS-defined files, and I know of no fields or flags there indicating that dissemination is to be suppressed. Is the only way I can know to download the .csv directory from the LUWS (not in timely sync with MLS) and try comparing the names and addresses? Not very practical, and error-prone, I fear.

Otherwise, I would have to rewrite the Families component, which is otherwise intended to mimic the data elements included in LUWS and be suitable for ward distribution, to be driven directly by that LUWS file. The home-teaching and visiting-teaching files, intended for limited distribution to ward, quorum and RS leaders, must be driven by the extracts..

danpass wrote:Although mentioned in this or other threads in this forum, your readme probably ought to mention that the administrator who extracts the data from MLS should remove confidential data from the csv files, particularly the membership.csv file. I suggest that you indicate which columns are needed by your scripts along with instruction to the MLS admin to delete all other columns before releasing the data.


I agree that would be wise, and I will document those fields. There is, as I recall, no way in the MLS user interface way to suppress fields from Membershiip.csv, as there is from PalmFamily.csv. So to do that reliably, someone would have to write yet another script to filter the fields, and that script would have to run on the clerlk's computer. Not that big a deal, but an additional complication.

danpass
Member
Posts: 491
Joined: Wed Jan 24, 2007 5:38 pm
Location: Oregon City, OR
Contact:

Postby danpass » Sat Jun 28, 2008 8:17 pm

boomerbubba wrote:I am in agreement with the intent of respecting members' wishes to be excluded, but I am not well educated about the mechanics of that. How precisely are those wishes executed within MLS? Does it only affect LUWS?

My inherent problem is that my application is driven by the data that is exported directly in those MLS-defined files, and I know of no fields or flags there indicating that dissemination is to be suppressed. Is the only way I can know to download the .csv directory from the LUWS (not in timely sync with MLS) and try comparing the names and addresses? Not very practical, and error-prone, I fear.

Otherwise, I would have to rewrite the Families component, which is otherwise intended to mimic the data elements included in LUWS and be suitable for ward distribution, to be driven directly by that LUWS file. The home-teaching and visiting-teaching files, intended for limited distribution to ward, quorum and RS leaders, must be driven by the extracts.


The LUWS exclusion is handled by a ward web admin through administration interface. He or She could be consulted. Other than that, as you suggest, it would need to be done by comparing the data from the two sources. The only time that comes to mind, where this situation ought to be considered is if there is an intent to upload the kml files to a ward or stake web site. I don't think that it is something you need to try to deal with in your code. It's just something that someone using your tools might want to consider if it applies to their situation.

RossEvans
Senior Member
Posts: 1346
Joined: Wed Jun 11, 2008 8:52 pm
Location: Austin TX
Contact:

Postby RossEvans » Sat Jun 28, 2008 8:40 pm

danpass wrote:The LUWS exclusion is handled by a ward web admin through administration interface. He or She could be consulted. Other than that, as you suggest, it would need to be done by comparing the data from the two sources. The only time that comes to mind, where this situation ought to be considered is if there is an intent to upload the kml files to a ward or stake web site. I don't think that it is something you need to try to deal with in your code. It's just something that someone using your tools might want to consider if it applies to their situation.


Actually, I am now quite concerned with this disjointed anomaly between LUWS and MLS. My intent is to produce a file suitable for dissemination to ward members generally, but I don't want to violate the spirit of commitments made to members who request "unlisted" status. The ward members generally have no more knowledge than I do about which families are supposed to be be treated that way.

Does this same anomaly apply to the "ward directory" reports produced by MLS and traditionally printed for ward distribution? That is, if a member requests to be excluded, is that member excluded only from LUWS but included in the MLS "ward directory" report?

If that is the case, then my application is doing no more harm than the MLS report is, but both would seem to violate the member's intent. If that is not the case, and the member is also excluded from the MLS ward directory, then I may be able to screen against that.

However, attempting to screen the MLS export data against the LUWS download data is inherently flawed. There is no reliable key for comparison, and even the content of the family name field does not match. Not to mention timing differences. Rather than attempt that kludge, I would sooner scrap my existing Family module and write another one driven only by LUWS.


Return to “General Clerk Discussions”

Who is online

Users browsing this forum: No registered users and 1 guest