RussellHltn wrote:You may want to try testing on Saturday morning. Seems to me that from the perspective of ISP load, it would be more like Sunday morning then the other times mentioned.
Yeah, we test a lot on Saturday morning, too. Worked great. I just mentioned Saturday night because we broadcast the adult session to 2 of the 3 other chapels and the broadcast went for almost 4 hours (an extra hour before and after) without a hiccup.
Same with Sunday morning. The priesthood leadership session broadcast (starting at 7 AM Sunday morning) went for just under 2 hours, then died. Once it died we never could reconnect to CHQ for longer than 30 seconds or so.
lajackson wrote:We are always told to test on the previous Sunday at the same time that the conference is scheduled. We have more trouble on Sundays that at anytime on Saturday.
But even Sundays are inconsistent. I suspect it may depend on how much stuff Hq is supporting on a given Sunday.
I've always been hesitant to test on Sundays because I don't want my test to interrupt / compete with someone else's live broadcast. If a bunch of people are testing on Sunday morning, you could easily add 50% - 100% more traffic to already strained servers.
I don't necessarily want to jump to the conclusion that this issue is due to an overload on the Church webcast servers. The reality is, we just completed a major upgrade to our server infrastructure. New outbound reflector servers were installed, and we upgraded the redundancy/failover system. This was completed several weeks ago, and from what we've observed so far, it's had a very positive effect on system reliability.
We also implemented a new webcast monitoring system that goes far beyond what we've ever had. Every 5 minutes, our system creates a new webcast in the same way that a user does - it creates a new stream, and we monitor every step of the process to make sure the stream is created properly, gets routed to the outbound reflectors, and streams out properly. If anything along the way fails, our engineers get notified.
That being said, I have our engineers checking the logs from this past weekend to see what the system load was and see if there were any errors. I will report back the results.
In the meantime, I would suggest there is a strong possibility that you were the victim of perhaps our most common webcasting issue. Users test during the week, and everything works great. Then they come to the actual meeting, and the webcast falls apart. The reason: meeting attendees come to the meeting packing smartphones, iPads, etc., and connect (either knowingly or unknowingly) to the building's wi-fi, and consume the bandwidth, which kills the webcast.
The only way to prevent that is to completely disable all other access to the building Internet - wired and wireless. Announcing from the pulpit and asking people nicely to shut off their phones just doesn't cut it.
Brian - you may or may not have disabled other Internet access; I didn't see that in your description. But it has been a common problem.
Kurt Olsen - Product Manager at Church HQ for Digital Presentation (Personal Video Conferencing, Meetinghouse Webcast, Conference Rooms, Video Conference Endpoints, Meetinghouse Digital Content)
ksolsen wrote:The only way to prevent that is to completely disable all other access to the building Internet - wired and wireless. Announcing from the pulpit and asking people nicely to shut off their phones just doesn't cut it.
The problem with the suggestion to disable wireless access to the building Internet is that the Church-managed Cisco 881W offers no reasonable way to turn off its wireless. We can disconnect any access points easily enough, but the wireless capability of the firewall itself is not under our control.
Kurt, I know it's not your area of responsibility, but since this very common issue has such an impact on webcasts, perhaps you could plead with your counterparts in the Meetinghouse Internet area to consider providing some way for us to disable the 881W's wireless for such events.
Questions that can benefit the larger community should be asked in a public forum, not a private message.
aebrown wrote:The problem with the suggestion to disable wireless access to the building Internet is that the Church-managed Cisco 881W offers no reasonable way to turn off its wireless. We can disconnect any access points easily enough, but the wireless capability of the firewall itself is not under our control.
Kurt, I know it's not your area of responsibility, but since this very common issue has such an impact on webcasts, perhaps you could plead with your counterparts in the Meetinghouse Internet area to consider providing some way for us to disable the 881W's wireless for such events.
In the, LDSTech Broadcast Aug 3, 2012 (~35 mins into it), Kurt said they were working on an official solution that would bypass or otherwise disable the firewall for a webcast, so maybe it's not too far off. In the mean time, I think a reasonable alternative (as he suggested) would be to allow the firewall to be bypassed for the express purpose of webcasting a stake conference.
So we can better help you, please edit your Profile to include your general location.
Mikerowaved wrote:In the mean time, I think a reasonable alternative (as he suggested) would be to allow the firewall to be bypassed for the express purpose of webcasting a stake conference.
However, the firewall is doing two things: Protecting the webcast server from the Internet, and as a router allowing more than one computer to be connected. You could substitute a home type router, but that's additional equipment.
I believe the suggestion has been to remove the antennas from the 881W. That limits the usable range to a very small value. Probably small enough to prevent problems.
Have you searched the Help Center? Try doing a Google search and adding "site:churchofjesuschrist.org/help" to the search criteria.
So we can better help you, please edit your Profile to include your general location.
Mikerowaved wrote:In the, LDSTech Broadcast Aug 3, 2012 (~35 mins into it), Kurt said they were working on an official solution that would bypass or otherwise disable the firewall for a webcast, so maybe it's not too far off. In the mean time, I think a reasonable alternative (as he suggested) would be to allow the firewall to be bypassed for the express purpose of webcasting a stake conference.
Actually, with my new role, the product we call Technology Manager is slowly migrating over to my responsibility. Technology Manager offers the ability to manage Internet access in the meetinghouses. We are working in the requirement to restrict Internet access so that you could turn off wireless and dedicate it solely to a webcast. I don't have an exact ETA on that feature yet, but we are driving toward it.
In the meantime, although Church IT security doesn't love this recommendation, I wouldn’t blame you if you physically bypassed the router only for the duration of the webcast. Then immediately reconnect it afterward.
Disconnecting the antennas can help, but it is not a full solution - it decreases signal strength, but the signal can still be available even with them disconnected.
Kurt Olsen - Product Manager at Church HQ for Digital Presentation (Personal Video Conferencing, Meetinghouse Webcast, Conference Rooms, Video Conference Endpoints, Meetinghouse Digital Content)
Kurt, sounds great. I would like to add one thing and that is to use the PVC for broadcasting. We successfully in Jun 2012 broadcasted our stake conference that way. We had no drops. Also, we could cut down the bandwidth by limiting the number of people we saw on the screen. There are still a few bugs to work out, but overall I found it to be great, more reliable and a much better option if we lose the Internet connection at the stake center so that we can broadcast it via phone.
ksolsen wrote:Brian - you may or may not have disabled other Internet access; I didn't see that in your description. But it has been a common problem.
This was actually the first broadcast where we DID turn off the wireless (actually we changed the password so no one could connect).
But bandwidth speed tests from the broadcast machine still showed 15M down and 2 M up, so nothing was consuming the bandwidth. Also, the problems started before the crowds really showed up, and the only thing that changed (as far as I can tell), was that once we got disconnected we could no longer reconnect for more than 30 seconds.
It's great that the infrastructure was just upgraded (I'm sure more and more of the stakes are using this, all on Sunday morning, which poses a real challenge for your IT team). I'd be interested to know if the logs show anything.
Absolutely. PVC can be a very good alternative to use for stake conferences. With its H.264 SVC technology, it is more tolerant of varying Internet conditions.
Kurt Olsen - Product Manager at Church HQ for Digital Presentation (Personal Video Conferencing, Meetinghouse Webcast, Conference Rooms, Video Conference Endpoints, Meetinghouse Digital Content)
