881W Issues

Discussions about Internet service providers (ISPs), the Meetinghouse Firewall, wired and wireless networking, usage, management, and support of Meetinghouse Internet
dromrell
New Member
Posts: 5
Joined: Sun Jan 06, 2013 12:26 pm

Re: 881W Issues

Postby dromrell » Sun Jan 06, 2013 1:01 pm

Answered my own question - The WIFI network failed again about 10 minutes after the Cisco router was rebooted.

rwardell
New Member
Posts: 7
Joined: Tue Dec 06, 2011 8:37 pm

Re: 881W Issues

Postby rwardell » Sun Jan 06, 2013 1:17 pm

Just adding my thoughts here on what we are seeing. When the building is empty network works flawlessly (wired and wirelessly). Teachers come and go testing and getting ready for their classes. for the last two weeks though as we have picked up the usage on the system when the building is crowded (Sunday meeting times) and lots of users we either have sporadic connections or none at all on the wireless side.

Is there any way at all for the local technology specialist to see the number of connections on a particular router. I can see the total number available and see that we have a 30 minute lease but cannot see how many leases are actually being utilizied. It would be really handy to see the DHCP table to know who and what is connected at any one time and possibly even be able to bounce some devices off the network if needed. A notification list would be great!

Clingmann
New Member
Posts: 16
Joined: Thu Dec 06, 2012 9:35 pm

Re: 881W Issues

Postby Clingmann » Sun Jan 06, 2013 2:39 pm

or how about being able to view the 881 device log . . .

If I had to guess, I would say the "controller" that is failing is the one that validates the WiFi passphrase. You'll notice it's usually on the join that people get the failure.

User avatar
johnshaw
Senior Member
Posts: 2155
Joined: Fri Jan 19, 2007 1:55 pm
Location: Syracuse, UT

Re: 881W Issues

Postby johnshaw » Sun Jan 06, 2013 2:47 pm

The Controller Architecture is likely not designed for the expected usage for the New Youth Curriculum. I hope they will be working day and night to balance out the controller situation. It would be nice if someone came on here and gave us an explanation, its not like it's a big secret or anything, and would likely benefit the STS to know what is hapening. It is possible that what we're doing in response makes the issue worse, and knowing information might help us work together to lessen the problem.
“A long habit of not thinking a thing wrong, gives it a superficial appearance of being right, and raises at first a formidable outcry in defense of custom.”
― Thomas Paine, Common Sense

tyranlewis
New Member
Posts: 7
Joined: Sat Feb 18, 2012 9:25 pm
Location: Dallas, TX Area

Re: 881W Issues

Postby tyranlewis » Sun Jan 06, 2013 3:29 pm

The LDS Tech person said that this is the second time this has happened. And that they will now try to figure out what is causing it. Hello?? How many times does it need to crash before they investigate. If my server / mail team told me after a crash of my servers that we need another time for them to crash for them to investigate?? I would have a few things to tell them. You investigate and and fix after the first time. Especially a global outage like this...

User avatar
Mikerowaved
Community Moderators
Posts: 3805
Joined: Sun Dec 23, 2007 12:56 am
Location: Layton, UT

Re: 881W Issues

Postby Mikerowaved » Sun Jan 06, 2013 6:30 pm

rwardell wrote:Is there any way at all for the local technology specialist to see the number of connections on a particular router.

There are actually several tools available. I use one called Fing for that type of visibility. It's 100% free, with versions available for the following devices:

    Microsoft Windows XP/Vista/7
    Mac OS X
    Linux (All major distros)
    Android smartphones and tablets running ver 2.2 or higher
    Apple iPhone/iPad/iTouch devices running version 4.2 or higher
    Kindle Fire tablet
    Cisco Cius tablet

Note: If CHQ has extended the IP Addressing range of your 881W, it may issue addresses that come from two or more different IP Address pools or ranges. Fing will only scope out the IP range it's currently connected to. To see information in another range, you would have to have a device attached with a network address in that range. This can sometimes be tricky to do, unless there is a static IP Address available for you to use in that range. (Don't use a static IP Address, unless you are SURE it's available.)
So we can better help you, please edit your Profile to include your general location.

gbekmezian
New Member
Posts: 21
Joined: Tue Dec 21, 2010 8:50 pm

Re: 881W Issues

Postby gbekmezian » Tue Jan 08, 2013 2:01 am

Hello tech community. I would like to begin by expressing my most sincere apology for the frustration the wireless connectivity issues in meeting houses has caused many of you. We are working very hard to understand the cause of the service interruptions and hope to prevent any widespread interruptions in the future. I have a request from the community as part of our effort to understand the scope of the incident from this past Sunday as well as trying to isolate the cause of the issues.

If your stake and/or ward experienced an outage this past Sunday and you DID NOT power cycle your wireless access points, I would like to know about it. Since the automatic response is always "reboot" there isn't much data available to us for root cause analysis. We are also still trying to understand the scope of the service interruption. Your cooperation and patience is greatly appreciated!

gbekmezian
New Member
Posts: 21
Joined: Tue Dec 21, 2010 8:50 pm

Re: 881W Issues

Postby gbekmezian » Tue Jan 08, 2013 2:19 am

JohnShaw wrote:The Controller Architecture is likely not designed for the expected usage for the New Youth Curriculum.

Actually, the "FlexConnect" architecture is designed for this type of use case. We did experience a controller failure last Sunday which doesn't quite explain the user impact. The technology is designed to keep functioning independently of the controllers for any access points already in operation.
I hope they will be working day and night to balance out the controller situation.

That is exactly what we are doing.
It would be nice if someone came on here and gave us an explanation, its not like it's a big secret or anything,

What would you like to know?
It is possible that what we're doing in response makes the issue worse, and knowing information might help us work together to lessen the problem.

At this point, I agree with you. Power cycling access points likely exacerbates the problem. There are two primary issues we are currently seeing. One is the exhaustion of IP addresses at a meeting house. Sometimes the solution is to assign additional address space to a meeting house, but a better solution is to reduce the length of the DHCP reservation time. If you call the GSC they should change the lease length to 30 minutes for you. Rebooting the 881w will clear the DHCP leases from the firewall's memory, though it is only a temporary fix.

The second issue has to do with a software failure we are still trying to find the root cause of. We are working diligently to do this, though it has proven to be difficult since we haven't collected enough data to determine the scope of the issue. We have engaged the vendor for support and will definitely escalate the case to the highest level if the work we do between now and Sunday hasn't solved the problem.

User avatar
johnshaw
Senior Member
Posts: 2155
Joined: Fri Jan 19, 2007 1:55 pm
Location: Syracuse, UT

Re: 881W Issues

Postby johnshaw » Tue Jan 08, 2013 6:15 am

gbekmezian wrote:JohnShaw said "It is possible that what we're doing in response makes the issue worse, and knowing information might help us work together to lessen the problem."

At this point, I agree with you. Power cycling access points likely exacerbates the problem. There are two primary issues we are currently seeing. One is the exhaustion of IP addresses at a meeting house. Sometimes the solution is to assign additional address space to a meeting house, but a better solution is to reduce the length of the DHCP reservation time. If you call the GSC they should change the lease length to 30 minutes for you. Rebooting the 881w will clear the DHCP leases from the firewall's memory, though it is only a temporary fix.


If this is the case, should instructions be sent out to all STS over the email list and an MLS message to clerks instructing them NOT to power cycle the firewalls if the event were to happen again on Sunday. If our default actions are resulting in your inability to figure out and fix the issue, it seems right to get some instructions out to us.

This is very helpful to have someone from the Church log in and interact with us. Thanks for doing so, remember, we are talking about 2 weeks in a row of issues (12/30 and 1/6), not just this past week. So comments on the board about the root cause diagnosis is already into its second incident that seems through observation to be the same event repeated.

You mentioned that the FlexConnect solution is designed to function in its current state, but also that a controller failure happened which seemed to impact us, so that seems a bit interesting, which is likely what has you all stumped as well :)

From your post you mention that there are instances where the demand has increased, based on the new come follow me videos (assumption) - and that adding an additional DHCP range to our firewalls and/or reducing our dhcp scope lease duration are 2 solutions to the problem?

Does lowering the lease duration reduce the need to add additional Scopes?

Should the church team consider pushing those changes out to everyone if you've identified it as a best practice? divide it up among the teams out there and just start knocking them out? (yes I know it's a manual process and there are a bunch of them). Your tools should be able to identify on Sundays which 881W's dhcp scopes are being consumed.

We're all in this together, and you have hundreds of STS always willing to help, the best way to have them help rather than hinder an issue is to communicate with them.

thanks again for jumping on the boards.
“A long habit of not thinking a thing wrong, gives it a superficial appearance of being right, and raises at first a formidable outcry in defense of custom.”

― Thomas Paine, Common Sense

User avatar
johnshaw
Senior Member
Posts: 2155
Joined: Fri Jan 19, 2007 1:55 pm
Location: Syracuse, UT

Re: 881W Issues

Postby johnshaw » Tue Jan 08, 2013 6:24 am

gbekmezian wrote:
JohnShaw wrote:
It would be nice if someone came on here and gave us an explanation, its not like it's a big secret or anything,

What would you like to know?


What happened?

Why did it happen?

What are the next steps to ensure it doesn't happen in the future?

basic root cause analysis info, but mainly the helpful actions or information we as STS need to know so can properly respond to and inform ecclesiastical leaders in our stakes.

For Instance, maybe a stake has a ward conference next week they are relying on broadcasting for, A Stake President might be doing PPI's over Video Conferencing, or a stake conference that is webcast. If we KNOW the incident is not resolved, we can start planning for next Sunday to provide an alternative to the internal Church Infrastructure (just a temporary fix to support the upcoming event). There is always the potential for unknown issues, but any known issues we should be able to account for and work around through planning. But there is large potential for a stake not to have been impacted, not have come to this board, not called into the GSD, so they have no way of knowing that a potential disaster awaits them next Sunday.
“A long habit of not thinking a thing wrong, gives it a superficial appearance of being right, and raises at first a formidable outcry in defense of custom.”

― Thomas Paine, Common Sense


Return to “Meetinghouse Internet”

Who is online

Users browsing this forum: No registered users and 1 guest