Saturday, December 31, 2016

Skype for Business Server Access Edge service does not start

Problem

You’ve noticed that the Skype for Business Server Access Edge service on your Skype for Business Server 2015 Edge server is stopped and the following error is thrown when you attempt to start it:

Windows could not start the Skype for Business Server Access Edge on Local Computer. For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to the service-specific error code -2146762487.

imageimage

Reviewing the event log displays the following errors:

Log Name: System

Source: Service Control Manager

Event ID: 7031

Level: Error

The Skype for Business Server Access Edge service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 180000 milliseconds: Restart the service.

image

Log Name: System

Source: Service Control Manager

Event ID: 7024

Level: Error

The Skype for Business Server Access Edge service terminated with service-specific error %%-2146762487.

image

Log Name: Lync Server

Source: LS Server

Event ID: 12303

Level: Error

The protocol stack reported a critical error: code 0x800B0109 (Configuration failure prevented the server from starting up). The service has to stop.

image

Log Name: Lync Server

Source: LS Server

Event ID: 12303

Level: Error

The protocol stack reported a critical error: code 0x800B0109 (CERT_E_UNTRUSTEDROOT). The service has to stop.

image

Log Name: Lync Server

Source: LS Protocol Stack

Event ID: 14623

Level: Error

A serious problem related to certificates is preventing Skype for Business Server from functioning.

Unable to use the certificate configured for the external edge of the Access Edge Server.

Error 0x800B0109(CERT_E_UNTRUSTEDROOT).

The certificate may have been deleted or may be invalid, or permissions are not set correctly.

Ensure that a valid certificate is present in the local computer certificate store. Also ensure that the server has sufficient privileges to access the store.

Cause: The Skype for Business Server failed to initialize with the configured certificate.

Resolution:

Review and correct the certificate configuration, then start the service again.

image

Log Name: Lync Server

Source: LS Protocol Stack

Event ID: 14397

Level: Error

A configured certificate could not be loaded from store. The serial number is attached for reference.

Extended Error Code: 0x800B0109(CERT_E_UNTRUSTEDROOT).

image

Clicking on the Details tab show the following:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

- <System>

<Provider Name="LS Protocol Stack" />

<EventID Qualifiers="33769">14397</EventID>

<Level>3</Level>

<Task>1001</Task>

<Keywords>0x80000000000000</Keywords>

<TimeCreated SystemTime="2016-12-30T01:27:45.000000000Z" />

<EventRecordID>154713</EventRecordID>

<Channel>Lync Server</Channel>

<Computer>svr-edge-01.ccs.int</Computer>

<Security />

</System>

- <EventData>

<Data>0x800B0109(CERT_E_UNTRUSTEDROOT)</Data>

<Binary>A6AC495DE63987EAE958F6506F58377D</Binary>

</EventData>

</Event>

image

One of the first troubleshooting steps I attempted was from the following blog post:

Attempting to follow the instructions provided by this blog post does not apply to your situation:

http://www.lyncexch.co.uk/lync-edge-january-2014-cu-update-issue/

However, using the following cmdlets to review the certificates’ serial numbers does not show a match for either:

  • A6AC495DE63987EAE958F6506F58377D
  • D77385F6056F859EAE78936ED594CA6A (reverse of the serial above)

Set-Location Cert:\LocalMachine\My

Get-ChildItem | FL

image

Get-ChildItem -Path 6224B3942798530F57A6F9BB560061BAF125DF1F | Format-List -Property *

image

**The serial for this certificate is 68000000BD4AC93CAEFE91A8BB0000000000BD

Get-ChildItem -Path 379944BB47EE3EE70E7ED9E5908041A5556F69CE | Format-List -Property *

image

**The serial for this certificate is 7D37586F50F658E9EA8739E65D49ACA6

Solution

As I’ve come across a similar problem in the past, I sort of had a feeling that this had to do with a certificate that was missing from the intermediate or root store of the Edge server.  To determine this, open the Certification Path of the certificate being used for the Edge interface:

image

Note that the issuing Certificate Authorities are:

  • GeoTrust Global CA
  • RapidSSL SHA256 CA

In this environment, the Root certificate GeoTrust Global CA was already in the Trusted Root Certification Authorities but the RapidSSL SHA256 CA was not in the Intermediate Certification Authorities:

image

I proceeded to obtain the issuing intermediate certificate via RapidSSL’s website:

https://knowledge.rapidssl.com/support/ssl-certificate-support/index?page=content&id=INFO1548

image

https://knowledge.rapidssl.com/support/ssl-certificate-support/index?page=content&actp=CROSSLINK&id=SO28616

image

Installed the certificate:

image

Then was able to successfully start the Skype for Business Server Access Edge service:

image

Friday, December 30, 2016

Dirsync export job in Synchronization Service Manager displays “InvalidSoftMatch” in the Export Errors

Problem

You’ve noticed that a newly created user account in your on premise Active Directory is not showing up in your Office 365 Admin center so you review the Operations menu in the Synchronization Service Manager and notice that the export job displays the error InvalidSoftMatch in the Export Errors window pane:

image

Opening the InvalidSoftMatch entry brings up the following Connector Space Object Properties Pending Export tab with information confirming that this is the missing user account:

image

Continuing to click on the Export Error tab displays the following information with a Detail button:

image

Clicking on the Detail button will display the following Error Information:

Unable to update this object because the following attributes associated with this object have values that may already be associated with another object in your local directory services: [ProxyAddresses SMTP:crussell@Contoso.com,smtp:crussell@ContosoReAG.mail.onmicrosoft.com,Mail crussell@Contoso.com;].  Correct or remove the duplicate values in your local directory.  Please refer to http://support.microsoft.com/kb/2647098 for more information on identifying objects with duplicate attribute values.

Tracking Id: 466344fe-a7c5-403e-8b0a-8621752ac178

image

You attempt to use the following PowerShell cmdlets via the WAAD (Windows Azure Active Directory) console to determine whether there is another account with the same smtp address:

Connect-MsolService

Get-MsolUser -all | Select DisplayName,ProxyAddresses | where-object {$_.ProxyAddresses -like "*crussell*"} | Format-Table -Wrap -Autosize

image

… but no results are returned.

Going through the steps outlined the in the KB:

Duplicate or invalid attributes prevent directory synchronization in Office 365
https://support.microsoft.com/en-us/kb/2647098

… and using the IdFix DirSync Error Remediation Tool (https://www.microsoft.com/en-ca/download/details.aspx?id=36832) does not list any references to the problematic account.

Solution

After trying all of the above without having any luck, I reread the contents in the following KB:

Duplicate or invalid attributes prevent directory synchronization in Office 365
https://support.microsoft.com/en-us/kb/2647098

… and noticed this:

All alias values in Office 365 must be unique for a given organization. Even if you have multiple unique suffixes after the at sign (@) in the Simple Mail Transfer Protocol (SMTP) address, all alias values must be unique.

Knowing that the user of the user account in question also had a pre-existing contact with an external SMTP email address, I began reviewing the properties of the existing contact in the Admin center:

image

Proceeded to click on the Edit Exchange settings link:

image

Which brought me to the Office 365 Exchange console of the contact object and it immediately became obvious that the problem was caused by the Alias of the exist contact (also configured as crussell):

image

Attempting to change the Alias would fail with:

image

error

The action ‘Set-MailContact’, ‘Alias,EmailAddresses’, can’t be performed on the object ‘Craig Russell’ because the object is being synchronized from your on-premises organization. This action should be performed on the object in your on-premises organization.

image

Attempting to delete the mail contact would throw the following error:

error

The action ‘Remove-MailContact’, ‘Identity’, can’t be performed on the object ‘Craig Russell’ because the object is being synchronized from your on-premises organization. This action should be performed on the object in your on-premises organization.

image

Having no luck with the GUI, I proceeded to review the Remove-MsolContact cmdlet:

https://docs.microsoft.com/en-us/powershell/msonline/v1/remove-msolcontact

Used the following Get-MsolContact cmdlet to export contact objects and their respective properties.  Used the find feature to locate the Craig Russell contact’s ObjectID:

image

Then proceeded to use the Remove-MsolContact cmdlet to delete the contact from the directory:

image

With the contact deleted, re-running the export job in the Synchronization Service Manager no longer displayed the InvalidSoftMatch in the Export Errors window pane:

image

Logging back onto the Office 365 Admin center console now displayed the user object.

Thursday, December 29, 2016

Removing GoToMeeting, GoToWebinar, GoToTraining icons in XenApp and XenDesktop 7.11

I’ve noticed that many of my clients have noticed that the GoToMeeting, GoToWebinar and GoToTraining icons can mysteriously appear when using the Citrix Receiver to connect to applications after upgrading their XenApp or XenDesktop from, say 7.6 to 7.11:

image

Administrators will know that earlier versions of StoreFront allowed you to disable these icons by clicking on Stores then the Integrate with Citrix Online link on the right but later versions have now removed this option.  To disable the icons shown above, navigate to Configure Store Settings:

image

Then in the Citrix Online Integration:

image

Uncheck the buttons:

image

Wednesday, December 21, 2016

Migrating Receive Connectors from Exchange 2010 to 2016 with native Exchange PowerShell cmdlets

I’ve been recently involved with a project where I had to assist a client with a Exchange 2010 to 2016 migration and one of the tasks I was assigned to do was to migrate the existing Exchange 2010 Servers receive connectors to the new Exchange 2016 servers:

image

I’ve come across this quite a few times in the past and even wrote a blog post 6 years ago:

How do I export/import Exchange 2007/2010 receive connectors’ allow relay IPs?
http://terenceluk.blogspot.com/2010/11/how-do-i-exportimport-exchange-20072010.html

It has been awhile since I’ve had to do this so I thought there must be a better way of doing this and what seemed to be the popular solution was this this PowerShell script:

COPY EXCHANGE SERVER RECEIVE CONNECTOR
https://www.granikos.eu/en/justcantgetenough/PostId/209/copy-exchange-server-receive-connector

Unfortunately, the script did not work for me as I would receive the following error when attempting to migrate the Exchange 2010 receive connectors to 2016:

[PS] C:\Scripts>.\Copy-ReceiveConnector.ps1 -SourceServer exhc01 -ConnectorName " Anonymous Relay Connector" -Targ
etServer bmexmb01 -MoveToFrontend -ResetBindings -DomainController dc01.domain.com

Security warning
Run only scripts that you trust. While scripts from the internet can be useful, this script can potentially harm your
computer. If you trust this script, use the Unblock-File cmdlet to allow the script to run without this warning
message. Do you want to run C:\Scripts\Copy-ReceiveConnector.ps1?
[D] Do not run  [R] Run once  [S] Suspend  [?] Help (default is "D"): r
At C:\Scripts\Copy-ReceiveConnector.ps1:116 char:13
+ </a>        <a href="/open-source" class="js-selected-navigation-item nav-item n ...
+             ~
The '<' operator is reserved for future use.
At C:\Scripts\Copy-ReceiveConnector.ps1:118 char:13
+ </a>        <a href="/business" class="js-selected-navigation-item nav-item nav- ...
+             ~
The '<' operator is reserved for future use.
At C:\Scripts\Copy-ReceiveConnector.ps1:120 char:13
+ </a>        <a href="/explore" class="js-selected-navigation-item nav-item nav-i ...
+             ~
The '<' operator is reserved for future use.
At C:\Scripts\Copy-ReceiveConnector.ps1:122 char:11
+ </a>      </nav>
+           ~
The '<' operator is reserved for future use.
At C:\Scripts\Copy-ReceiveConnector.ps1:1736 char:225
+ ... hidden" value="&#x2713;" /></div>
+                    ~
The ampersand (&) character is not allowed. The & operator is reserved for future use; wrap an ampersand in double
quotation marks ("&") to pass it as part of a string.
At C:\Scripts\Copy-ReceiveConnector.ps1:1767 char:11
+       <li>&copy; 2016 <span title="0.10192s from github-fe158-cp1-prd.iad.github ...
+           ~
The ampersand (&) character is not allowed. The & operator is reserved for future use; wrap an ampersand in double
quotation marks ("&") to pass it as part of a string.
At C:\Scripts\Copy-ReceiveConnector.ps1:1767 char:23
+       <li>&copy; 2016 <span title="0.10192s from github-fe158-cp1-prd.iad.github ...
+                       ~
The '<' operator is reserved for future use.
At C:\Scripts\Copy-ReceiveConnector.ps1:1767 char:29
+       <li>&copy; 2016 <span title="0.10192s from github-fe158-cp1-prd.iad.github ...
+                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Unexpected token 'title="0.10192s from github-fe158-cp1-prd.iad.github.net">GitHub</span>' in expression or statement.
At C:\Scripts\Copy-ReceiveConnector.ps1:1767 char:100
+ ... ">GitHub</span>, Inc.</li>
+                    ~
Missing argument in parameter list.
At C:\Scripts\Copy-ReceiveConnector.ps1:1786 char:14
+       You can't perform that action at this time.
+              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The string is missing the terminator: '.
    + CategoryInfo          : ParserError: (:) [], ParseException
    + FullyQualifiedErrorId : RedirectionNotSupported

[PS] C:\Scripts>

image

I did a bit of troubleshooting and searching but could not find a solution so decided to fall back on the same approach in my older blog post using native Exchange PowerShell cmdlets and a bit of editing of the information in notepad.

Before I begin demonstrating the PowerShell cmdlet and switches to use, let’s begin by reviewing the parameters we’ll be configuring for a receive connector by looking at the options in the EAC:

General Options

image

Note that the parameters available to be configured in the screenshot above are:

  • Name
  • Connector status
  • Protocol logging level
  • Maximum receive message size (MB)
  • Maximum local hop count
  • Maximum hop count

These parameters map to the following PowerShell switches:

Parameter Switch

Name

Name

Connector status

Enabled

Protocol logging level

ProtocolLoggingLevel

Maximum receive message size (MB)

MaxMessageSize

Maximum local hop count

MaxLocalHopCount

Maximum hop count

MaxHopCount

Security Options

image

Note that the parameters available to be configured in the screenshot above are:

  • Authentication
  • Permission Groups

These parameters map to the following PowerShell switches:

Parameter Switch Options

Authentication

AuthMechanism

  • Transport Layer Security (TLS)
    • Enable domain security (mutual Auth TLS)
  • Basic authentication
    • Offer basic authentication only after starting TLS
  • Integrated Windows authentication
  • Externally secured (for example, with IPsec)

Permission Groups

PermissionGroups

  • Exchange servers
  • Legacy Exchange servers
  • Partners
    Exchange users
  • Anonymous users

Scoping Options

image

Note that the parameters available to be configured in the screenshot above are:

  • Remote network settings
  • Network adapter bindings
  • FQDN

These parameters map to the following PowerShell switches:

Parameter Switch

Remote network settings

RemoteIPRanges

Network adapter bindings

Bindings

 

FQDN FQDN

Step #1 – Retrieve and Export Receive Connector Configuration

With the configuration parameters outlined above, the first step for migrating the receive connectors to the new Exchange server is to use the Get-ReceiveConnector to export the receive connectors’ information.  The following is the cmdlet with the switches required:

Get-ReceiveConnector -Server <sourceExchangeServerName> | Select Identity,Name,Enabled,ProtocolLoggingLevel,MaxMessageSize,MaxLocalHopCount,MaxHopCount,AuthMechanism,PermissionGroups,RemoteIPRanges,Bindings,FQDN

The cmdlet above will output something similar to the following:

image

Notice that the RemoteIPRanges configuration output gets truncated when the list has more than 16 entries which means if the list has less than 16, you’re set to go but if you have more then you’ll have to execute the following before using the Get-ReceiveConnect cmdlet:

$FormatEnumerationLimit =-1

Get-ReceiveConnector -Server <sourceExchangeServerName> | Select Identity,Name,Enabled,ProtocolLoggingLevel,MaxMessageSize,MaxLocalHopCount,MaxHopCount,AuthMechanism,PermissionGroups,RemoteIPRanges,Bindings,FQDN

image

The cmdlets above will list all of the IPs:

image

Now that we have the cmdlets to export all of the information, proceed to pipe the output to a text file:

$FormatEnumerationLimit =-1

Get-ReceiveConnector -Server <sourceExchangeServerName> | Select Identity,Name,Enabled,ProtocolLoggingLevel,MaxMessageSize,MaxLocalHopCount,MaxHopCount,AuthMechanism,PermissionGroups,RemoteIPRanges,Bindings,FQDN > C:\ReceiveConnectorsExport.txt

image

For the parameter options of the Get-ReceiveConnector cmdlet, refer to the following TechNet article: https://technet.microsoft.com/en-us/library/aa998618(v=exchg.160).aspx

Step #2 – Extract configuration and create new Receive Connector

With the information in the text file we created above, proceed to extract the information, format them properly and use the following cmdlet to create the new receive connector:

New-ReceiveConnector -Name “<NameOfReceiveConnector” -Enabled $true –ProtocolLoggingLevel <verbose or none> –MaxMessageSize <numberMB> -MaxLocalHopCount <number> –MaxHopCount <number> –AuthMechanism <authentication> –PermissionGroups <groups> -RemoteIPRanges x.x.x.x,.x.x.x.x -Bindings 0.0.0.0:25 -FQDN <FQDN for connector> -Server <targetServerName> -TransportRole FrontendTransport

Items to consider for the New-ReceiveConnector cmdlet:

  • You have to remove custom if it is in the PermissionsGroups configuration output
  • You have to add the TransportRole switch with FrontendTransport parameter at the end for Exchange 2013 and 2016 target servers
  • The FQDN may need to change depending on the name used
  • Adjust the bindings IP address if the source configuration uses a specific IP address that is not configured on the target server

For the parameter options of the New-ReceiveConnector cmdlet, refer to the following TechNet article: https://technet.microsoft.com/en-us/library/bb125139(v=exchg.160).aspx

The follow is an example of the cmdlet with parameters:

New-ReceiveConnector -Name “Anonymous Relay Connector” -Enabled $true -ProtocolLoggingLevel verbose -MaxMessageSize 10MB -MaxLocalHopCount 8 -MaxHopCount 60 -AuthMechanism Tls -PermissionGroups AnonymousUsers -RemoteIPRanges 10.21.1.110,10.5.1.3,10.22.1.212,10.22.1.41,10.22.1.68,10.22.1.210,10.22.1.211,10.22.1.157,10.22.1.10,10.22.1.159,10.22.1.25,192.168.60.28,10.34.30.222,10.20.1.75,10.22.1.164,10.23.0.79,10.22.1.37,10.2.1.37,10.22.1.233,10.23.0.19,10.22.1.49,10.34.30.26,10.21.1.142,10.20.1.64,10.43.3.140,192.168.60.172,10.20.1.50,10.21.1.101,10.20.1.83,10.21.1.117,10.23.0.60,192.168.60.158,10.20.1.175,10.21.1.174,192.168.60.162,10.31.30.22,10.23.0.54,10.22.1.51,10.21.1.41,10.20.1.189,10.20.1.162,10.5.8.242,10.21.1.130,10.21.1.10,10.34.10.30,10.5.3.21,10.20.1.174,10.7.3.11,10.7.3.4,10.7.3.8,10.7.3.3,10.20.1.47,10.21.1.111,10.21.1.92,10.22.1.131,10.21.1.26,10.21.1.24,10.21.1.25,10.21.1.23,10.21.1.22,10.34.30.221,10.21.1.51,10.6.3.92,10.6.3.91,10.6.3.53,192.168.60.56,192.168.60.102,10.34.170.60,10.21.1.49,10.34.30.17,10.21.1.48,10.20.1.126,10.20.1.30,10.34.10.11,10.34.10.20,10.34.10.21,10.34.10.10,10.21.1.60,10.21.1.88,192.168.60.58,10.21.1.54,10.20.1.72,10.43.3.80,10.43.3.81,10.21.1.52,10.20.1.181,192.168.60.48,10.20.7.177,192.168.60.183,10.3.2.10,10.20.2.15,192.168.60.127,192.168.60.205,192.168.60.174,192.168.60.104,192.168.60.173,192.168.60.191,10.3.3.203,192.168.60.169,192.168.60.108,192.168.60.55,192.168.170.60,192.168.60.170,192.168.170.30,192.168.60.106,192.168.70.152,192.168.60.198,192.168.60.189,10.3.3.200,192.168.60.185,192.168.70.148,192.168.60.9,192.168.170.125,192.168.170.128,192.168.60.91,192.168.60.148,192.168.170.26,192.168.60.135,192.168.60.24,192.168.60.21,192.168.60.19,192.168.160.136,192.168.60.57,192.168.170.147,192.168.60.8,192.168.60.160,192.168.173.10,192.168.60.124,192.168.160.129,192.168.20.28,192.168.200.0/24,192.168.100.0/24 -Bindings 0.0.0.0:25 -FQDN EXHC01.domain.com -Server bmexmb01 -TransportRole FrontendTransport

image

Monday, December 19, 2016

“Skype for Business Server 2015, Backup Service central management backup module has backup data that never gets imported by backup pool.” is logged on primary server referencing the backup server

Problem

You’ve recently failed over the CMS and pool from a primary Skype for Business Server 2015 to a backup server then failed the services back to the primary server but started noticing the following errors logged on the primary active server with the contents referencing the backup server:

Log Name: Lync Server

Source: LS Backup Service

Event ID: 4098

Level: Error

Skype for Business Server 2015, Backup Service central management backup module has backup data that never gets imported by backup pool.

Backup data file: \\drlyncstd01.domain.com\LyncShare\2-BackupService-1\BackupStore\CentralMgmt\CMSMaster\Data\Backup.zip
Cause: Import issue in the backup pool. Please check event log of Skype for Business Server 2015, Backup Service in the backup pool for more information.
Resolution:
Fix import issue in the backup pool

image

image

Note that the content in the error log references the backup / disaster recovery server but the event is logged on the primary active server.

The backup / disaster recovery server also has the Skype for Business Server Master Replicator Agent service stopped.  You can start the service but it will stop after a few minutes:

image

Solution

I’ve noticed this happening in most SfB environments that have experienced a planned or unplanned failover of the CMS and/or pool.  To correct the issue, simply launch the Skype for Business Server 2015 – Deployment Wizard on the backup / disaster recovery server, navigate into the Install or Update Skype for Business Server System menu:

image

… and then run both the:

  • Install Local Configuration Store
  • Setup or Remove Skype for Business Server Components

image

Once the above have successfully completed, the Skype for Business Server Master Replicator Agent service should now start and stay running:

image

The event ID 4098 error should no longer be logged:

image

… and an event ID 4099 information log should be written indicating:

Import issue in the backup pool was fixed. Skype for Business Server 2015, Backup Service central management backup module export operation continues now.

image

image