LCR not working 4/10/22

[andersenr]

It is 2:16 pm MST and the finance section of LCR still isn’t working.

[hendeli]

Same issue. Any resolutions?

[jdlessley]

Wait, so are we supposed to make a deposit even if we can’t use LCR/F? That’s the first I’ve heard that. Where do we find the guidance and process for that?

Help Center article Donations during System Outage.

[jelgadeans]

LCR finance not working for us in Rigby, Idaho. Tried for over an hour to bring it up and it just spins on load. This is seriously wasting a lot of time of bishopric members trying to get tithing entered not to mention time away from families. This has been an issue ever since we started using the system during the beta. Please upgrade server capacity!

[TimothySyoen]

Wait, so are we supposed to make a deposit even if we can’t use LCR/F? That’s the first I’ve heard that. Where do we find the guidance and process for that?

Help Center article Donations during System Outage.

Thank you for posting that JD! Will definitely be keeping the link you posted for any similar times in the future.

I've been tangling with LCR Finance for the better part of the last hour, glad to know it's not just us here in Mesa, AZ.

[davesudweeks]

Please upgrade server capacity!

It has been shared in other threads that server capacity is not the issue (and if it were, it would have been fixed long ago).

[plott]

You say it has nothing to do with the servers - but when people ask you to improve availability they don't always just mean the physical hardware. There is a lot you can do to improve availability. 1) increase bandwidth 2) redundant and distributed hardware and processes to handle requests. 3). Properly sized, benchmarked and stress tested databases and webserver utility(full stack). 4) Networking - this includes being able to handle and address issues such a denial of service attacks.

Just saying you can't increase availability because you can't add more hardware is a basic cop-out


Based upon the symptoms, it definitely seems that you are having issues addressing the # of web requests on Sundays. Something seems to be crashing early in the morning when usage increases and then when you do finally get it back up the requests are fulfilled so slowly it seems like you have some serious bottlenecks in the full stack, storage, or networking.

There is a reason websites that experience heavy traffic don't always go down. Proper testing and preparation makes a lot of difference. Most often these issues arise with new websites because the weren't designed and tested to handle the realistic web traffic expected. When engineering these systems there is also a necessity in ensuring a safety factor to handle spikes.

[russellhltn]

We don't know the details, but from what has been said, I don't think there's a simple solution. I think some technology has reached it's limits and can no longer scale. Replacement is challenging.

If I had to take a wild guess, I'd say a backend database has saturated. Replacing it would require re-writing all the apps that talk to it. Not easy for an organization that runs close to 24/7 and has always seemed a bit understaffed in the development team.

We can only hope the recent problems are hastening the work to do what needs to be done and things will be much better than they have been.

[jonesrk]

You say it has nothing to do with the servers - but when people ask you to improve availability they don't always just mean the physical hardware. There is a lot you can do to improve availability. 1) increase bandwidth 2) redundant and distributed hardware and processes to handle requests. 3). Properly sized, benchmarked and stress tested databases and webserver utility(full stack). 4) Networking - this includes being able to handle and address issues such a denial of service attacks.

You are correct that there are many parts to this that can play a role in causing load issues. One of the key parts in this is the Membership database that was initially designed about 30 years ago and has been bolted onto for a long time. That system was not designed with web access in mind and is not structured to be able to scale well. We began a few years ago to build the replacement system for that. We are getting there, but with so many things tied into it, it has been a slower process than we would like. We are getting there in many areas but are not as far as we would like to be.

[lajackson]

Another thing that helps a little is to remember that the very people how are trying to fix these issues also have to deal with these challenges themselves each Sunday as they try to process their own financial and membership transactions as clerks in their own wards. It is a small consolation, but I am sure it does keep them motivated, as well.

I am just disappointed that what was becoming the MLS backup has also gone away.