Meetinghouse Internet Problems

[drepouille]

Right. I have decided to ignore all those red pop-up messages that say the status of the firewall could not be determined. If I can drill down into the firewall, and see the WAPs and connected devices, then I know all is well.
I just wish the usage statistics would work as they once did.

[yarrgh]

TM this morning and it was kicking all sorts of errors and didn't show a correct status.

This is because of a constraint with the vendor API. With the amount of people checking their firewall statuses, we are exceeding the rate limit. We've been working on a solution to better handle this so that we don't exceed it so much, but it isn't quite ready. And now because of the outage, it is really affecting everyone. TM is almost unusable at times because of the amount of traffic and everyone constantly refreshing trying to get a successful status return for their firewalls.

If we get an error (rate limit, etc.) we come back with a unknown status. This doesn't necessarily mean that there is an issue with your firewalls. Just that TM doesn't know the status. If you absolutely can't get the online status of your firewalls and you absolutely need to know before you head on site (travel times, etc.), you can PM me with your serial numbers and I can check them manually.

[russellhltn]

Well, it's back. Two of the new firewalls in my stake are showing "warning". One had never been reset and all the APs are showing off-line. The other had been "paper clipped" before, and the APs are showing on-line. I'll need to call to find out what's going on there. That one has had "warnings" in the past due to a IP address collision, but I thought it was fixed.

[russellhltn]

Apparently a problem in TM where you'll get a false status. So, there's still a small kink in the system.

[mfmohlma]

Apparently a problem in TM where you'll get a false status. So, there's still a small kink in the system.

Yup. See the post right before yours.

[russellhltn]

Apparently a problem in TM where you'll get a false status. So, there's still a small kink in the system.

Yup. See the post right before yours.

I think that's different. When TM is overloaded, you'll get an actual pop-up error message. This is a bit more subtle. No pop-up error message, but a "warning" status. Since everything was fine Sunday, I thought it was all resolved.

[lajackson]

Since everything was fine Sunday, I thought it was all resolved.

Wasn't there a post a while back that said there were limitations on how TM is getting its information? I think they are still trying to resolve that.

[russellhltn]

Wasn't there a post a while back that said there were limitations on how TM is getting its information? I think they are still trying to resolve that.

Right. But what's I've seen when that happens is a pop-up message on my screen. IIRC, at first it's just a thing on the right side, but then there's an actual popup I have to click though. It's not subtle at all.

I suppose it's possible they've removed that "feature" and let it fail more quietly as it is rather annoying to get one for every firewall in the stake every time you go to the dashboard.

[lajackson]

it is rather annoying to get one for every firewall in the stake every time you go to the dashboard.

I wonder if it has anything to do with the number of firewalls you have in your stake. Perhaps a lot (6-10 or more) are triggering the error, but five or less or not. Or maybe there is some other magic number in between where TM can fetch the information without raising too much of a ruckus.

[russellhltn]

As I understand it, the overload problem is the total number of users hitting TM at the same time, not individual TM users. The new firewalls are controlled by "the cloud" and all those requests hit upon a single cloud provider. When the number of requests gets too high, it gets throttled and hurts the TM users.

Because of the notice sent out, a lot of people were hitting TM at the same time to check on the status of their systems.