Gospel Library Mobile Content

The Gospel Library Mobile Content (GLMC) project is intended to automate the process of transforming ldsWebML into a more suitable mobile format. As there are several mobile platforms supported and each platform has it's own capabilities we recognize that there may be a need for multiple formats. This document is written in present tense regardless of whether what is being described is currently implemented or not.

Project Details

Project Resources


This project is ongoing and is expected to continue improving over the next few months.


99% of the content published on Gospel Library is provided by the Curriculum Department. This content is referred to as LDS XML. LDS XML is given to the ICS (Internet Communication Systems) Content and Media team to be transformed for displaying this content online ( / Mobile). This content is referred to as ldsWebML. Then this content is passed to the Mobile Development team where we do our own transformation into the various mobile formats.

We get content as a book. Book of Mormon, June 2011 Ensign, True to the Faith, are all examples of books that we could get new or updated files for. For example, if you have checked out the GospelLibraryMobileContent GIT project (this is a rather large project so beware) navigate into the ldsWebML/english/scriptures/bofm directory. Notice that there are sub-folders representing the books within the Book of Mormon as well as some xml files in this directory.

The XML file that represents the book is called 03990_000_bofm_000.xml. This file is our starting point for transformation and points to all of the included files that represent the entire book.


  1. Content is formatted by the Curriculum Department with a focus on publications (Ensign, Liahona, or Printed Manuals) called ldsxml.
  2. Ldsxml content is passed on to the Content and Media team.
  3. The Content and Media team transforms the content into ldsWebML.
    1. Standardizes the file XML tag names into more Web friendly tag names.
    2. Strips some unecessary XML tags.
    3. Adds URI's to the book, sections of the book, and paragraphs/verses.
      1. URI's are important as they are what is currently used to uniquely identify content allowing us to share annotations (Notes, Highlights, Cross-references, etc.) between and Mobile devices.
  4. ldsWebML is passed on to the Mobile team and this is where this project really begins.
  5. In gospel-library/scripts there are several useful scripts for automating the transformation of ldsWebML to our mobile format(s). In the python scripts (py extension) if you open the file for editing you will see that most have a fairly well documented 'USAGE' section. If the script is not mentioned here it is either because it is included in one of the other main scripts or because it is no longer used.
    1. Using xqsync/ we can pull content from the Content and Media team. Unfortunately this requires VPN access into the church network. If the content is already checked in to the gospel-library/ldsWebML directory then this step is unnecessary.
      1. Using we copy the changes that were downloaded using xqsync into the ldsWebML directory structures. As it is copying it also checks the local sqlite database to see if it should be marked as needs_updating.
    2. Using we can convert one or more ldsWebML books into a format suitable for both Android and iPhone as well as other things (other actions are documented below).
      1. This uses the to simplify the HTML as well as transform it if necessary.
    3. Using we can load our local sqlite database (found in syncData directory) with the catalog of books that are available currently in production.
    4. Once the catalog is locally available you can run ' -scan' to scan the local directories and see if there is anything available to be uploaded.
    5. New Content: Connect to the database in the syncData directory using sqlite3.
      1. Tip: To exit sqlite3 type '.exit'. To see the database schema type '.schema'
      2. If you run the query 'select * from languages;' you will see a list of the available languages defined on the Gospel Library Admin Console. Take note of the id's associated with the languages as we usually work with one language at a time.
      3. If you run the query 'select * from folder where languageid = 1;' you will see a list of the folders available for that language.
      4. If you run the query 'select id, catalog_id, folder_id, uri, name from book where languageid=1 and catalog_id=0;' you will see a list of books availble for English (languageid=1) where the book has not been uploaded to the glweb service (catalog_id=0). These are the potential new books available for upload.
        1. At this point we need to determine what should be uploaded.
        2. Tip: The URI is always in English and should not differ. This means that the URI for the 2011 June Ensign will be /ensign/2011/06 for all languages. This is helpful in knowing what the book actually is and what folder to upload a book into.
        3. This step may require folders be created on glweb admin. If it does and you do not have access then contact the Development Lead listed below in Communications to get those created.
        4. Once we have determined which books should be uploaded into which folder we can run a script on the local sqlite to update the folder_id. Look at gospel-library/scripts/content/english/_contentUpdate.sql for an example of how to update the folder_id based upon URI's and folder names.
        5. After folder_id's have been assigned then run 'python -upload' This requires that you have valid credentials for the glweb admin console. If you do not have access you may need to contact the Development Lead listed below in Communications.
    6. Update Content: Use xqsync with the xs:dayTimeDuration option to specify to only download the changes that have happened over a specific time. See gospel-library/scripts/xqsync/ file for an example.
      1. Make sure you have run to create the local database.
      2. Using we copy the changes that were downloaded using xqsync into the ldsWebML directory structures. As it is copying it also checks the local sqlite database to see if it should be marked as needs_updating.
      3. Follow the steps contained above in the New Content section to see if there is anything new to be uploaded.


There are several tips that we have learned while doing the transformation.

  1. If things are not looking right you can always delete the database in gospel-library/script/syncData/* then run 'python' again.

Server functionality

When we upload the content to the server it does the following.

  • If new upload: Create a new book entry with the values passed in during upload.
  • If update of existing book: Update the values associated with the book with the values passed in during upload.
  • Save the uploaded file to a local directory as well as copy that file to the CDN location.
  • Records the last modified date of that catalog. Remember a catalog is specific to a Language Id and a Platform ID.

Client Configuration

instructions for setting up the Perl configuration are

1.5) Install Command Line Tools (Recent XCode versions only) (Thank you to Tom Marchioro for informing me about this step.)

Older versions of XCode installed the command line tools (which are required to properly install CPAN modules) by default, but apparently newer ones do not. To check whether you have the command line tools already installed, run the following from the Terminal:

$ which make

This command checks the system for the “make” tool. If it spits out something like /usr/bin/make you’re golden and can skip ahead to Step 2. If you just get a new prompt and no output, you’ll need to install the tools:

Launch XCode and bring up the Preferences panel. Click on the Downloads tab Click to install the Command Line Tools If you like, you can run which make again to confirm that everything’s installed correctly.

2) Configure CPAN. $ sudo perl -MCPAN -e shell

perl> o conf init

This will prompt you for some settings. You can accept the defaults for almost everything (just hit “return”). The two things you must fill in are the path to make (which should be /usr/bin/make or the value returned when you run which make from the command line) and your choice of CPAN mirrors (which you actually choose don’t really matter, but it won’t let you finish until you select at least one). If you use a proxy or a very restrictive firewall, you may have to configure those settings as well.

If you skip Step 2, you may get errors about make being unavailable.

3) Upgrade CPAN $ sudo perl -MCPAN -e 'install Bundle::CPAN'

Don’t forget the sudo, or it’ll fail with permissions errors, probably when doing something relatively unimportant like installing man files.

This will spend a long time downloading, testing, and compiling various files and dependencies. Bear with it. It will prompt you a few times about dependencies. You probably want to enter “yes”. I agreed to everything it asked me, and everything turned out fine. YMMV of course. If everything installs properly, it’ll give you an “OK” at the end.

4) Install your modules. For each module…. $ sudo perl -MCPAN -e 'install Bundle::Name'


$ sudo perl -MCPAN -e 'install Module::Name'

This will install the module and its dependencies. Nice, eh? Again, don’t forget the sudo.

The first time you run this after upgrading CPAN, it may prompt you to configure again (see Step 2). If you accept its offer to try to configure itself automatically, it may just run through everything without a problem.

There are a couple of potential pitfalls with specific modules (such as the LWP::UserAgent / HEAD issue), but most have workarounds, and I haven’t run into anything that wasn’t easily recoverable.

We need to install the following as well.

sudo perl -MCPAN -e 'install +YAML'

sudo perl -MCPAN -e 'install SOAP::Lite'

sudo perl -MCPAN -e 'install IO::Socket::SSL'

sudo perl -MCPAN -e 'install Mozilla::CA'

Content Changes

It is not recommended that we change/fix the content that is pulled from the Content and Media team. Any issues found should be emailed to the Development Lead listed below in Communications

What can community do to help us

As you can see the workflow documented above is quite convoluted with many steps and different scripts that need to be run. We could use some help doing the following.

  • Simplify the automated process.
  • Testing the process to make sure that the content is transformed correctly. You can run 'python -noCompression bofm' to create GeneratedBooks/Book.of.Mormon.zbook that is actually just a sqlite database. You can then connect to the sqlite and perform queries on the node table to get an output of the content. Compare this with the original ldsWebML files to ensure data integrity and sound HTML formatting.
  • Need to update the php files in the glweb service enabling us to hit a service to generate the catalog.json resources and copying them over the to appropriate locations on the CDN. Request access to this SVN project from the Development Lead listed below in Communications.
  • Need to update the php files in the glweb service to check the uploaded files into SVN as well as copy them over the CDN. Request access to this SVN project from the Development Lead listed below in Communications.
  • Be creative... if you see a need bring it up to us. We may not have thought of it.


Development Lead: Nathan Dickamore -

This page was last modified on 15 July 2013, at 21:33.

Note: Content found in this wiki may not always reflect official Church information. See Terms of Use.