So recently I had a customer come to me looking for help with Office 365 Multi-Geo. They were looking at Telstra Calling for Office365 but their tenant is homed in the States. As such they had expressed concerns about the performance of Telstra calling for Office365 with media going to the States and back.
Just so my non-Aussie followers can follow along, TCO365 is our Microsoft Calling Plan equivalent. It’s not a Direct Route based solution as many of the hosted Direct Route solutions are. You can read more about that over here
What’s so bad about having media go to America and back anyway?
Voice and Video are both realtime applications. This means that users expect to interact with the other end in “Real Time” So having a high amount of latency in the call can really affect the user experience, remember we are trying to get the other party as close to “being there” as possible.
Now delay isn’t an issue when the communication is one way, there is no indication that an event happened some time ago. But as soon as a conversation involves 2 or more parties it gets a little more complicated.
Believe it or not, Humans communicate in a similar way to legacy Ethernet networks. Using our own variant of the now rarely used CSMA/CD protocol. We are (usually) taught this as children. With the key part of it being that humans typically wait approximately 100ms before recognising a gap in conversation (CSMA), then waiting a random period to see if the original speaker or another person starts talking (CA) and both restarting the process if we speak over each other (CD)
And this is the crux
of the issue. Introducing additional delay in a voice stream doesn’t cause any technical issues. The sound and video is still
conveyed using the technology. But the human part has difficulty coping with
the extra delay.
See, our brains are incredibly good at finding a point to butt-in to a conversation and we start processing a “gap” in as short as 100ms (we don’t react that quickly, but we still pick up on it). Not to mention that if we hear ourselves with a delay of over 100ms we can find that echo distracting instead of just, well echo.
So for example an average round trip time from the land down
under to the states is around about 200ms, human reaction time is still limited
to the 200-300ms range.. So we should be good right? Well not really.
Let’s say I’m talking about something to a co-worker over a long latency link. When we first start the conversation we may find the conversation awkward because we are talking, waiting a bit to see if the other person has something to say and then talking again.
If you start processing that “gap” you left in the conversation before the other end has even had the chance to her you stop speaking. Your likely to start thinking you can talk again before the other person has head the gap, thought about what they are going to say and engage their slow brain to say it.
I used to do a lot of satellite comms in my early VoIP days and a 600-800 ms delay was common. Conversations over satellite were terrible. Because you would have to adjust your conversations to compensate for the long latency.
Whilst you’re not doing this consciously on a call with 200ms latency, your brain is still running through these processes and adjusting your experience. When you know you’re talking with someone far away you naturally adjust, but when you call your local business about that stationary order? Not so much.
How did we traditionally address this in Skype4B on premises.
Very badly.
If the user couldn’t send media direct from one endpoint to another. It would use the users home edge server.
So if you had 2 staff members working overseas for a conference and they called each other. Media would go back to Australia and back even if they were just in different hotel rooms.
The only real way around it without deploying another pool or SBA was to use a local SBC with media bypass.
How Office365 addresses this
Office365 and
especially Teams leverage stuff we have been using in the SIP and Skype4B world
for years. ICE, STUN and TURN. Using Transport
Relays to get media from one place to another as quickly and efficiently
as possible.
These Transport Relays are exactly that, they simply shuffle packets from one
place to another. They don’t do any transcoding, look at the packets or the
contents (they are encrypted anyway) The only packet manipulation they will
perform is if one of the endpoints can only communicate via TCP it will send
the TCP data via UDP to the UDP capable endpoint and vice versa.
Teams Transport Relays differ from Skype4B Media Relays in that they are reachable via Anycast addresses as well as unique IP addresses. Instead of just globally unique addresses.
Anycast? Sounds like a steaming service.
To breakdown Anycast a bit in case you haven’t seen them before. Anycast addresses allow multiple machines across the internet to have the same IP address. This is done via some BGP trickery that I’m not getting into here, but basically your ISP is aware of all the instances of that same address and will always send to the one with the lowest routing cost.
Once the node has
been found, hosts will typically switch to using the unicast (traditional) IP address when using TCP so that
connections aren’t broken when a routing change occurs at your ISP.
(This is likely another reason UDP is so important in Teams. Because if we are connection-less, we don’t care what physical Transport Relay we talk to. As long as it routes the packets to the endpoint)
Should a node fail, it stops “advertising” that it has the Anycast address and your ISP moves the route to the next closes host.
(Note: These are
just example IP addresses plucked from the Office365 IP ranges for
Teams/S4B)
Pretty neat huh? Did you know you’re probably using an Anycast IP already? Well known DNS services like Google and Cloudflare run on anycast addresses like 8.8.8.8 and 1.1.1.1 respectively.
Okay, but what about getting onto the Azure network?
Hangon, let me get my megaphone out.
*ahem*
USE LOCAL INTERNET BREAKOUT.
*smooths shirt*
See each one of those little orange dots? That’s an Azure edge node. Basically where traffic destined for Office365 gets off the internet and into Microsoft’s Azure network that’s been designed to carry this kind of data.
Unless your ISP is running on a potato they should be peering with Microsoft directly or via one of the peering networks so you should only be a few hops away from the Azure network.
I’m going to pull a bit of my old ISP BGP nerdery here and point out the main AS that Microsoft use for their Teams peers is 8075 (https://www.peeringdb.com/asn/8075) and with a quick bit of Looking Glass magic, I can see exactly how Telstra routes their traffic to Microsoft (Most ISP’s offer a “Looking Glass” page, even the one I used to work at)
As you can see, with the addresses listed (a Teams media relay) it’s just a few hops away.. The first AS is Telstra and the second Microsoft
Even the not for
profit IX peering network has state based connectivity into Microsoft.. So any
ISP can buy an unmetered 10gb of peering bandwidth into IX for as little as $350 a month! (I remember buying 100Mbit at these
prices)
How does Teams deal with Multi-Geo then?
From a media and calling perspective it doesn’t. From a data at rest perspective it simply uses the underlying service’s Multi-Geo. (AFIAK The Microsoft Azure Chat Service is not included in this)
As Teams bolts onto the other Office365 services that benefit from Multi-geo such as OneDrive and SharePoint, it is multi-geo aware in a similar way to the Office 365 app launcher portal.office.com. You can @ mention (or atless mention) users in any Multi-Geo region, Users data is stored in their local region for personal chats (their OneDrive for Business) and in the relevant SharePoint Group site for the Team they are participating in.
So what is Office 365 Multi-Geo for anyway?
Simple. As this isn’t Futurama and we aren’t governed by Richard Nixon and the Government of Earth. It’s a solution for large multinational companies to adhere to their local data sovereignty laws. Perhaps they are working on some product or solution the government doesn’t want stored overseas in the event of some international upset. What about who owns the data and the laws that regulate that data, it’s hard to apply your nations rules on data hosted overseas. Even if your citizens did create it.
Remember that I mentioned before that users can store data in the Team they are participating in? In a Multi-Geo environment the user that created the team sets the team’s geographical location based on the users Multi-Geo Location setting. If this is a concern information barriers may be the way to go, but that’s a different topic entirely.
You can find more about Multi-Geo here
https://docs.microsoft.com/en-us/office365/enterprise/multi-geo-user-experience
https://products.office.com/en-us/business/multi-geo-capabilities
What’s the point of all this then?
The long and the short of it is, No. You don’t need to deploy Multi-Geo for Voice, meetings or video in Microsoft Teams. There are plenty of network engineers smarter than I am working to get your media to the relevant location as quickly as possible.
You might however need to deploy Multi-Geo if you have data soverentiy to compliance issues which mean your data at rest must be stored in country.
I know this has been a bit of a longer one, so if you liked it. Let me know.
References used:
https://tomtalks.blog/2019/06/where-in-the-world-will-my-microsoft-teams-meeting-by-hosted/
https://myignite.techcommunity.microsoft.com/sessions/65521?source=TechCommunity (BRK4016 Ignite 2018)
https://myignite.techcommunity.microsoft.com/sessions/65507 (BRK4013 Ignite 2018)
Transport Relays in Skype4B Online and Teams
https://azure.microsoft.com/en-us/blog/how-microsoft-builds-its-fast-and-reliable-global-network
https://tomtalks.blog/2018/05/office-365-multi-geo-is-here-what-about-microsoft-teams-and-skype-for-business/