Marrying Cloud Video with On-Premises Calling 2/2

This blog is working under the assumption that the following on-premises pieces have already been built out and are working independently of the WebEx cloud piece:

• Cisco Unified Communications Manager solution on-premises
• On-premises PSTN connectivity via a Cisco Voice Gateway or CUBE

This blog also assumes that the WebEx piece has already been provisioned and that Video endpoints are already registered to the cloud.

Part 2 of this blog adds some details around the pieces that added to what’s listed above to pull this together. Here’s the high-level list of technologies mentioned again.

• WebEx Control Hub
• Cisco CUCM
• Cisco Expressways
• Firewalls
• NAT
• DNS
• Certificates
• Cisco Directory Connector software

Please read all of the Cisco guides thoroughly before attempting design or installation. With that said here’s a little bit of information about each technology with my tech notes.

The curve balls I ran into in the process are in orange text. 

WebEx Control Hub

The WebEx Control Hub is the partner/customer configuration touchpoint for the WebEx cloud. Below is a high-level list of most of the things configuring in the WebEx Control Hub for Hybrid Calling.

• Created a custom SIP subdomain (for example joshualearn.com)

• “Verify” the subdomain via a DNS text record with a special hash value provided in the WebEx Control Hub
• Optionally “claim” the subdomain with a TAC case. (If the subdomain is claimed then any new WebEx Teams account with that email domain has to be created in the official Company organization)
• Call Connect
• It gets enabled pointed to the SRV record for the MTLS connection to the Expressway-E For example, _sips._tcp.sipmtls.joshualearn.com (It’s also possible to use an IP and port in the case of a quick POC)
• Upload the expressway certs to WebEx (if they’re self-signed)
• Active Hybrid Call Service
• Add a default SIP destination for Hybrid Call Connect Service
• Enable Hybrid Calling for the user in the WebEx cloud. (The directory URI of the Spark Remote Destination should match the user's mail id in CUCM which should also match the user-id in the WebEx control hub

Cisco CUCM

The Cisco CUCM should already be in place but there a few things that to be validated and or remediated to start and there a few things that need to be configured specifically to the integration build-out.

General Prerequisites

• The CUCM database replication needs to be built out using FQDNs. The Expressway will connect to the CUCMs using that same value and need a FQDN that can be resolved via DNS. This caught me off guard the first time and required additional planning to update the system to FQDN entries and to re-establish the database replication between CUCMs.

• Cluster Fully Qualified Domain Name in the Enterprise Parameters needs to be configured to something unique. For example cluster1.joshualearn.com

New build out for Hybrid Calling

• The CUCM needs a service account with AXL permissions (which is pretty standard for most applications)

• Cisco CUCM needs a SIP trunk built out to the Expressway C server
• Cisco CUCM needs SIP route patterns to send calls to WebEx.
• *.calls.webex.com

• *.rooms.webex.com

• Configure the Spark remote device
• Configure a Spark Remote Destination for the user

Cisco Expressway

The Cisco Expressways is a bit of Swiss Army Knife for Cisco when it comes to additional software required for Hybrid Calling. Most of the specific connector software runs on the Expressway-C Connector host. There is also a transit pair required to create the secure transport between CUCM on the WebEx cloud.

Cisco Expressway-C Services Host

• Cisco recommends that the Expressway-C Services Host is updated to the latest version of code. This will allow for interoperability with the WebEx Control Hub. This is also the reason why it’s helpful to have the Services host on its own dedicated Expressway. (Some customers have very small deployments and are tempted to use the same Expressway-C as the Connector host and the transit host. However, these can cause additional challenges with keeping up the latest Expressway code on the C and E. If the Expressway-C Connector host is separate than the transit pair for the Expressways can be updated on a different interval.

• The Management Connector is installed to establish connectivity to the WebEx Control Hub from the Expressway

• The Call Connector is installed to create the data hook between the CUCM and the WebEx control hub

• Linking the Call Connector to CUCM pulls the IP address or the FQDN of the CUCM. If the system was defined using a hostname in CUCM the connection will fail. This is an issue I ran once. For whatever reason, the folks that deployed the initial CUCM decided to use hostnames. The Expressway-C Connector host could not resolve the hostname and therefore could not connect the CUCM.

Cisco Expressway Traversal Pair

The traversal pair is basically a way for the SIP traffic to traverse from the Cisco CUCM on-premises environment to the WebEx Control Hub and cloud registered video endpoints. The expressway traversal hosts can be dedicated or shared with existing MRA and or B2B expressways. Refer to the Expressway guides for details on how to spin up and license them.

Cisco Expressway-C Traversal Host

• The Expressway-C host sits on the internal voice server network.

Cisco Expressway-E Traversal Host (with 2 NICs)

• The Expressway-E ideally sits in a firewall environment with a transit DMZ and a service DMZ. However, I have also seen it deployed with one NIC in a service DMZ and the other NIC on the inside network.

• Create an MTLS zone between the Expressway-E and the WebEx cloud.

• Follow the guide to configure zones and search rules on the Expressways.

• One caveat I ran into once was with using the automatic WebEx DNS zone. It ended up causing an error message to pop up. I don’t remember any more specifically what the message was. It may have been that is was looking for additional SIP ports to be available instead of just the MTLS SIP port 5062. In any case, I manually built this rule and I didn’t receive any errors.
• Another caveat I ran into was the tool to validate the connection to the WebEx control hub was bugged out and didn't support MTLS. That warning can be ignored. (At least until Cisco updates that portion of the code). Bug ID: 

Firewall

• Two firewall DMZs are recommended by Cisco however, some customers have opted to implement one and put the “transport DMZ NIC” on the inside network.

• Firewall policies
• Open up port 5062 on the firewall for traffic towards the Expressway-E (for the WebEx Control Hub)

NAT

• NAT the “Service DMZ” interface of the Expressway-E to a public IP address

DNS

• A record – create an A record for the Expressways. For example expe.us1.joshualearn.com.

• SRV record – create an MTLS SIP service record. For example _sips._tcp.sipmtls.joshualearn.com. 300 IN SRV 5 10 5062 expe.us1.joshualearn.com.

Certificates

• Internal certificates are allowable if the Expressway-E is only hosting the Hybrid Calling, however, the WebEx Control Hub needs to trust the issuing CA chain. If the Expressway-E is also hosting MRA and or B2B then a publicly signed certificate is required on the Expressway-E

Have you implemented this before, have thoughts about trying it out, or do you just like to leave comments? If so, please leave one below.

Marrying Cloud Video with On-Premises Calling 1/2

Cisco's current line of Video endpoints (Room Kit, Room Kit Plus, SX, DX, MX, and WebEx Board) can be registered to the WebEx Calling Cloud for signaling and SIP URI dialing. This allows a video endpoint to call another endpoint or call into a WebEx meeting using a URI.

(A URI looks like an email address. For example, I might set up a URI as dx80@joshualearn.com for folks to reach me on a Cisco DX-80. The URIs on WebEx are actually a bit longer. One might look more like dx80@joshualearn.rooms.webex.com on WebEx. Most of the time folks don't type out the URIs so that's not usually a concern.)

So what happens when you want to dial a PSTN telephone number?  How do we associate a PSTN telephone number to a video device?  Previously, the only option was to turn up a SIP trunk with a telco provider within the WebEx Control Hub.  Then one would order new DIDs, assign a DID, and assign an internal number to a video endpoint.   That process involves starting a service contract with a telco provider that has an established relationship with Cisco WebEx. Cisco has termed this type of company as a "Cisco Preferred Media Provider".

While that is still an option, some folks have asked how they can bring their own PSTN by integrating with their existing on-premises Cisco Collaboration investment.  Before there was not an option.  Now those folks are in luck.  Utilizing what's known as Cisco WebEx Hybrid Call Service the Cloud Video environment can be married to On-Premises Calling environment.

What's involved to make that work? I'll write a follow-up post with more detail.  In the meantime, here is a high-level list of what's under the hood.

• Cisco WebEx Cloud registered video endpoints
• Cisco Unified Communications Manager solution on-premises
• On-premises PSTN connectivity via a Cisco Voice Gateway or CUBE
• Cisco Expressways
• Cisco Expressway-C Services Host
• Cisco Expressway-C Traversal Host
• Cisco Expressway-E Traversal Host (with 2 NICs)
• Firewalls
• 2 firewalls are recommended, however, I have also seen it designed with 1.
• Firewall policies
• NAT
• DNS
• A records
• SRV records
• Certificates
• Internal certificates are allowable however the WebEx Control Hub needs to trust the issuing CA chain.

Part 2 of this blog series will include more of the detail on some of the specifics related to the "how-to" design and deploy the solution.

If you have tried this already, have thoughts about trying this, or just like to leave comments, please leave one below.

Why's the mainline forwarded to a cell phone!?! (Cellphone Carrier plays error after 60 seconds)

Here's a "fun" one.

Symptom:
I received intermittent reports/questions about the main line, "being forwarded to a cell phone".  I know there's no reason why I would have forwarded the mainline to a cell phone.

Troubleshooting:
I checked Cisco Communications Manager and confirmed the main line wasn't forwarded to another number.  Then I began to call the mainline from a land line & a cell phone but could not to reproduce the reported problem.  During the tests when an operator wasn't available after 90 seconds my call was routed to the auto attendant as designed.  I checked the logs for the reported problem call and found a normal call clearing from the PSTN.  Sadly, I wasn't any closer to finding where the end user's disconnect was.

Luckily, one of the reporting persons then sent me a video of an example problem call to the main line on her mobile phone.  Her  mobile phone rang for right around 60 seconds and then played the following message.

"
Welcome to Verizon wireless.  The wireless customer you called is not available at this time  Please try your call again later.  Announcement 1 switch [redacted]
"

This helped explain the confusion around why users were reporting the main line was being forwarded to a cell phone.  The Verizon error states that the "wireless customer" is not available.

I validated that the reporting user was indeed using a Verizon mobile phone.  Armed with that information I repeated the test from a couple of other Verizon mobile phones and was able to consistently recreate the issue.

Issue:
Verizon Wireless pulled back the call at 60 seconds and played the following error message:

"
Welcome to Verizon wireless.  The wireless customer you called is not available at this time  Please try your call again later.  Announcement 1 switch [redacted]
"

Work Around:
Modified the timers on the Cisco Communications Manager.  Previously the system was configured to route calls to the auto attendant after 90 seconds.  I discussed the issue with the customer and we decided to change the value to 30 seconds.  Now, when an operator doesn't pick up the call within 30 seconds, the call is routed to the auto-attendant instead of being disconnected by our friends at Verizon.

Where's my Caller-ID Calling Name Man? (PRI Inbound Calling Name in subsequent FACILITY)

It's been a long minute since I've run into this particular scenario.  Unfortunately, I didn't have the solution in my notes so I reached out to my old pal google for some inspiration.

We just had a new telco company install a voice PRI with the intention of replacing our existing telco company's PRI at one of our remote sites.  After installing and testing the new voice PRI we realized that the inbound calling name was not displayed on the phones.

In this particular scenario the setup looked something like this:

PRI -----ISDN-----> Cisco ISR-G2 VG -----SIP-----> CUCM -----SCCP-----> Cisco IP Phone

We ran a "debug isdn q931" to troubleshoot the problem.  Upon inspection we found that the calling party name was not being sent in the initial ISDN setup message.  In fact, it indicated that the calling name would be send in a "subsequent FACILITY message".

First, the voice gateway won't process the calling name in a subsequent Facility messages by default. The fix for that is to apply the following to the d channel of the PRI:

interface Serial X/X/X:23
 isdn supp-service name calling

Secondly, there is a problem in this scenario with the voice gateway sending a SIP invite to Communications Manager right after it gets the ISDN setup message.  Since the initial ISDN setup message doesn't have the calling name the SIP invite will also be lacking the calling name. When we tested at this point the phones were showing "From pending" as the calling name. The work around for this issue is to introduce a delay before the voice gateway sends the SIP invite to CUCM. Each scenario may require a different amount of time. One can look at the debugs to find out how long it takes between the time the initial ISDN setup message is received and when the ISDN facility message with the name is received. In this particular case we tuned the delay to 500 milliseconds and it seems to work well. The command to delay the setup for 500 milliseconds is.

sip-ua
 timers buffer-invite 500

For more information, Cisco also has a bug id CSCup91440 that references using a buffer-invite on individual dial peers.

Below are some samples of what the debugs looked like on a typical inbound telco call and the telco that sends the calling name in a later message.

Standard Telco "Before Debug"
(The actual telephone numbers have been changed to protect the innocent.)

507963: Jan 22 09:53:13.269: ISDN Se0/2/0:23 Q931: RX <- SETUP pd = 8  callref = 0x14D4
        Bearer Capability i = 0x8090A2
                Standard = CCITT
                Transfer Capability = Speech 
                Transfer Mode = Circuit
                Transfer Rate = 64 kbit/s
        Channel ID i = 0xA98382
                Exclusive, Channel 2
        Facility i = 0x9F8B0100A11702010D020100800F4368696361676F202020202020494C
                Protocol Profile =  Networking Extensions
                0xA11702010D020100800F4368696361676F202020202020494C
                Component = Invoke component
                        Invoke Id = 13
                        Operation = CallingName
                                Name Presentation Allowed Extended
                                Name = Joshua Learn
        Display i = 'Joshua Learn'
        Calling Party Number i = 0x2180, '8005551212'
                Plan:ISDN, Type:National
        Called Party Number i = 0x80, '8885551212'
                Plan:ISDN, Type:National
507964: Jan 22 09:53:13.269: ISDN Se0/2/0:23 Q931: Received SETUP  callref = 0x94D4 callID = 0x13E4 switch = primary-ni interface = User 


Subsequent Facility Message Telco "After"
(The actual telephone numbers have been changed to protect the innocent.)

040306: Jan 22 07:49:27.801 PST: ISDN Se0/2/0:23 Q931: RX <- SETUP pd = 8  callref = 0x1A22
        Bearer Capability i = 0x8090A2
                Standard = CCITT
                Transfer Capability = Speech
                Transfer Mode = Circuit
                Transfer Rate = 64 kbit/s
        Channel ID i = 0xA98381
                Exclusive, Channel 1
        Facility i = 0x9F8B0100A10F02016A06072A8648CE1500040A0100
                Protocol Profile =  Networking Extensions
                0xA10F02016A06072A8648CE1500040A0100
                Component = Invoke component
                        Invoke Id = 106
                        Operation = InformationFollowing (calling_name)
                                Name information in subsequent FACILITY message
        Progress Ind i = 0x8281 - Call not end-to-end ISDN, may have in-band info
        Calling Party Number i = 0x2183, '8005551212'
                Plan:ISDN, Type:National
        Called Party Number i = 0xA1, '8885551212'
                Plan:ISDN, Type:National
040307: Jan 22 07:49:27.801 PST: ISDN Se0/2/0:23 Q931: Received SETUP  callref = 0x9A22 callID = 0x1D81 switch = primary-ni interface = User
040308: Jan 22 07:49:27.805 PST: ISDN Se0/2/0:23 Q931: TX -> CALL_PROC pd = 8  callref = 0x9A22
        Channel ID i = 0xA98381
                Exclusive, Channel 1
040309: Jan 22 07:49:27.848 PST: ISDN Se0/2/0:23 Q931: RX <- FACILITY pd = 8  callref = 0x1A22
        Facility i = 0x9F8B0100A11702016B020100800F332054524143452020202020202020
                Protocol Profile =  Networking Extensions
                0xA11702016B020100800F332054524143452020202020202020
                Component = Invoke component
                        Invoke Id = 107
                        Operation = CallingName
                                Name Presentation Allowed Extended
                                Name = Joshua Learn