Citrix XenDesktop introduced a number of new concepts and processes for desktop administrators. The goal of this session is to demystify these concepts and provide a tactical approach to deployment and troubleshooting of a XenDesktop environment. In this session we will demonstrate the core configuration that is required, and also cover proven troubleshooting approaches to the top three problems we see in customer deployments.
Good afternoon my name is Ramon Scott and I am a Lead Escalation Engineer on the Citrix Escalation Team Today my presentation be on Configuring and Troubleshooting XenDesktop Sites.
let's begin by looking at the high level XenDesktop database and services architecture
We will start by looking at the database
XenDesktop supported databases are:SQL 2008 SP1, 2008 R2 and higher including express editions.The database schema has a full relational schema; complete with tables views and stored procedures there is a single database for the core productand there aremultiple schemas inside the databaseschemas map to windows services running on the broker
the setup process will depend on the environment What does this mean ?If there is a single administrator account used for SQL and XenDesktop then this admin will generate the schema from the console and the console in turnwill connect to the SQL database and execute the script to create the database . <.>Once this script has successfully executed the broker will connect to the database and verify that it is operationalIn case where their are separate XenDesktop and SQL admin account. the XenDesktop admin will generate the schema from the console, this schema will be exported to a SQL script that should be provided to the SQL admin. Next, the SQL admin will connect to the SQL server console and execute the script to create the database . Once this script has successfully executed the broker will connect to the database and verify that it is operational
Database accessThe runtime access performed by the XenDesktop DDCs on the database uses the following security model:In the environment where the controller and the SQL database are housed on the same Server the broker will connect to the database using its local network service accounts .NT AUTHORITY /NETWORK SERVICE On the other hand If the broker and SQL database are on separate servers, the broker will connect to using itscomputer accounts : “DOMAIN\MACHINE$”there is one SQL login for each brokerThis account login has a restricted permission thatmeans the broken does not have rights to change the schemaThe XD database contains a set of pre-configured DB roles which have detailed execute/select permissions hung off them. Each DDC has a dedicated user within the XD database that is a member of the above roles. Each DDC accesses the DB server through its AD machine account which requires it to have a login created for it, and for that login to be mapped to the associated user in the XD DB. The login does not need to be a member of any server-level roles.´The XenDesktop 5 services access the database using their computer account logins (domain\machine$, or „NT AUTHORITY\NETWORK SERVICE‟ if database is located on a controller ( i.e. SQL Express)
Now discuss our high availability options the Booker is critically dependent on the database however if there is a database failure existing connections will not be impacted but creating new connections and reconnecting to desktops willthat means a database failure equals a broker failure the supported high availability options are SQL mirroring ,virtual machine high availability and SQL clustering
Here is a table that Maps the XenDesktop services to the available database schemas
to perform basic health check on a XD site you can use the XDDBDiag tool
We will now transition and review the services architecture
The Machine creation service is responsible for the creation and provisioning activities for VMs and master ImagesMachine Identity Service, this service is responsible for the management of the Disks attached to the VMsAD Identity service, this service is responsible for the maintenance and creation of the AD computer accountsThe services combine to make-up the Machine Creation ServicesBroker service, this service is responsible for VDA registration, Power Management, license enforcement and resource allocation host service , this service manages the hypervisor connections and resourcesConfiguration Service, this service provides directory services metadata storage and security These two services make up the Infrastructure ServicesAll six services maintaining their own separate connection to the backend SQL database So what does this mean for you?Well, when there is an issue with Expanding catalogs , the personality of machines computer accounts lockouts, the troubleshooting efforts will be focused on the Machine creation servicesIf your issue is related to the hypervisor connection storage or its resources, Site configuration or errors in service communication your focus should be directed at the Infrastructure ServicesAnd for issues with registration, licensing, power management your efforts will be focused on the Broker ServiceDesktop studio is the management console used to configure the site and it leverages PowerShell and WCF typically on port 80Desktop director is a web based portal that can be used to the support and helpdesk teams to monitor and troubleshooting system issues before they become system-critical while at the same time Quickly and seamlessly perform crucial support tasks for their end users including view Performance statistics via WinRMWindows Communication Foundation (or WCF), previously known as "Indigo", is a runtime and a set of APIs (application programming interface) in the .NET Framework for building connected, service-oriented applicationsSOA: Service-oriented architectureRef:http://en.wikipedia.org/wiki/Service-oriented_architectureEach service instance reads and writes to the SQL database periodically using connectionless ADO.net.PoSH – PowerShellWCF – Windows Communication FoundationWinRM – Windows Remote Management
You can also receive the current status of any of the core services from the XenDesktop PowerShell prompt by running the the respective command from this list as you can see the syntax is quite intuitive Let’s look at the example in the first row‘to get the status of the AD identity service , you simply execute the get ‘dash’ acct service status
We can now review the concept of Machine Creation
In XenDesktop 5.6 and higher support , Seven virtual desktop models are supported we have existing which leverages virtual machines created outside of XenDesktop .These next three are created by desktop through MCSFor a dedicated catalog , the image is cloned and provided to multiple user that will have the ability to persist their changes going forward we then have pooled that create a cloned image that is then referenced as a single base image ; The virtual machines then saves any changes from the base image to a volatile diff disk that is discarded on rebootThen there is pooled with personal vdisk , this allows pooled machines the facility to save change to a separate disk that will persist with them on rebootNext there is streamed, this catalog leverages Citrix PVS server to stream a non-persistent image to the VM and all changes the image are lost on rebootand lastly we have streamed with Pvds , the steamed images the facility to save change to a separate disk that will persist with them on reboot
under the MCS options, A pooled catalog can be either Randomly assigned or statically Assigned and then with Dedicated catalogs you can have Pre-Assigned or assigned to a user on first usePooled with PVD operated like a static pool as the same machine is proved to the users in addition to there unique PVDThe dedicated model can be pre-assigned to a user or assigned on first use and they will retain the assignment going forwardAnd Now For The two PVS options , You can either steam to a virtual to physical desktop Where as steamed with PVD, allows you toonly steam to Virtual Machines Ref:Random in that a user gets a new pooled image or the static option in which the same images is provided to the users after reboots.
Machine creation service catalogs comprise of three disks, the Master base disk shared among all VMs in the catalog, and for each Virtual desktop, a diff disk and an identity diskThe Diff, This is what the user sees as Drive C:\And the identity disk this is hidden from the users view and maintains the machines personality configuration. This scheme is replicated for each disk that is created.
For Machine creation with Personal Vdisk , we have the base, and VM dif and id disk, however there is an addition personal vdisk auto-created by copying the pvd template from the base VM. This disk is 10 GB by default and has a 50/50 split for application data and user dataThis part is hidden from user and Merged with the Diff Disk and the users accesses its information as drive C:\ eg application data
PVD stream catalogs have the Streams vdisk, the write cache and the Base VM.The Streamed vDisk is what the user sees as Drive C:\And the Write Cache is visible file on another disk, typically D:\
Streamed with PVD has the base image and for each Virtual desktop there is a streamed vdisk and write cache however it also has the personal vdisk attached . The PVD is the same as in the MCS configuration, I.e.., defaults to 10 GB, there’s a 50/50 split , the application data is hidden and the User data is seen as the as P:\
You may be interested in also learning what are some of the common issues They are:Hypervisor communicationDomain permissionsPreviously failed attempts still present in databaseTheHost Connection configured with incorrect storage repositorythe host configured with and unsupported Naming convention
So What happened when after you just sold your business unit on the concept of moving to XenDesktop and they request 500 new desktops for tomorrow morning and the wizard Fails ?What logs do we need ? wellas previously mentioned for issue with machine creation you need the machine creation services logs from the three servicesYou may also collect the logs from the desktop studio to see what command were run and what errors we actually returnedAt this point all other logs can be ignored
When troubleshooting an issue I recommend the following MethodologyUnderstand issue historyWhat changed?When did is StartAny specific images it happens more frequently with?Verify configuration, error logs and alertsGather and review service log data of issuesCompare collected data to a working environment
Citrix provides many methods to enable and collect logs; and I wanted to provide one that was a little less knownYou can enable the service logs by locating the service executable and launching with the parameter dash logfile and then the valid location for the log fileHere we go and enable the machine creation service logsNext the AD identify service logs and lastly the Machine identity serviceReference for log enablingC:\Program Files\Citrix\MachineCreation\Service>Citrix.MachineCreation.SdkWcfEndpoint.exe -logfile c:\xdlogs\MCS-PVSvm.logC:\Program Files\Citrix\ADIdentity\Service>Citrix.ADIdentity.SdkWcfEndpoint.exe -logfile c:\xdlogs\AD.logC:\Program Files\Citrix\Host\Service>Citrix.Host.SdkWcfEndpoint.exe -logfile c:\xdlogs\host.logC:\Program Files\Citrix\MachineCreation\Service>Citrix.MachineCreation.SdkWcfEndpoint.exe -logfile c:\xdlogs\MC.logC:\Program Files\Citrix\MachineIdentity\Service>Citrix.MachineIdentity.SdkWcfEndpoint.exe -logfile c:\xdlogs\mi.log
great, We have done quite a lot so far, now lets see how it applies with a case study
This case study is actually something seen in my lab when rebuilding my lab for a new case.This is a New Deployment with the Latest Hotfixes , I utilized account with Full Administrative access to the domain and hypervisor . This environment worked before it was rebuilt
So I started by reviewing the Desktop Studio logsThe search terms I users were the [Time of Issue].* Fail | Error | Exception | DeniedIt quickly returned the error seen in the console “Failed to copy all master images to all of the Hosts. No machines have been added to the Catalog.”Ok It tell me I failed however I need to dig further
So I then review the Machine Creation Service logI used the same search terms [Time of Issue].* Fail | Error | Exception | DeniedIt returned with a from interesting errorsFirst it reported : Failed to copy disk. Reason : SR_HAS_NO_PBDSThen it how and Managedexception with the same errorNext it indicated the job I created was concluding with a disk consolidation failed errorAnd finally it reported the error seen in the console and Desktop studio logsSo what do you do next, that have a detailed error, we can use it and conduct some research via forums , search engines etc.
So the issue was a misconfigurationThe main clue is that messageFailed to copy disk Reason : SR_HAS_NO_PBDS So research shows that SRs are Storage Repositories and PBD are Physical Block Devices implies that the sr does not have access to the disk, so we check and found thatHypervisor Connection’s did not include correct storage for the Master ImageThe Broker could not communicate with the correct storage location and therefore we were not able to create the base master image . When creating the hypervisor connection we followed the defaults of the wizard and did not specify the location that we utilized from images.
Desktop Service is startedLooks up list of DDC from local registry or from personality.ini fileLooks up the computer accounts in Active DirectorySelects one DDC from the DDCs list at random and then initiates a connection through WCFDDC receives or rejects connectionIf DDC rejects or does not respond, service wait for the timeout then selects another DDC at randomIf DDC receives connection, it looks up the VDA computer accountChecks that computer account (SID) isin published assignmentInitiates WCF Test connection to VDAQueries the VDA stateSets configuration and policiesVDA is marked as registeredReturns success for registrationVDA starts heartbeat ping to DDC
CTX123278Xd Ping report networking information of the machinePerforms time checks against the domainRetrieve the current Users informationRef Info:Information and status of Network Interfaces and Network settings. (Console Only)Performs DNS lookup and reverse lookup on the IP address of the device.Information on Time synchronization and time check for Kerberos Authentication (Console Only)User information for login User (Console Only)Including User details, Authentication type used, Group Membership.Machine information (Console Only)Environment information (Computer Name, operating system version, Domain) Domain membership verification (Membership = Verified, SID:S-X-X-XX-XXXXXXXXXX-XXXXXXXXXX-XXXXXXXXXX-XXXXX [OK])Information on XenDesktop Services (Windows Communication Foundation Endpoints) installed and confirms if each installed service is responsive. (Console Only)Displays information on the Windows Firewall installed on the VDA and checks if the important ports are configured correctly.Queries the local event log to check for known events that are related to XenDekstop.Provides client bandwidth and response time information from the VDA to the client.
Bidirectional network tests, verifying that telnet is possible bidirectional, verify that the services are started and are the listening on the correct portThen disable any firewallsWe then check the time against the Doman and between the ddc and VDA next we verify the registry entry for the list of ddcsIf after these check we still haven’t found the root cause , then we enable service logging on the Vda and broker
Its now time to review the second Case Study.
DDc not AutorizedPolicy found that was needed but not enabledResolution to explicitly enable access
When troubleshooting Vda Issues the best place to start is asking the simple questions what happenedWhat did the user see ?was it just a generic error ? Did the error indicate the actual causeDid the receiver start and disappear without and errorOr did the receiver launch and then return and error
When trouble shooing a launch issue should start by reviewing the event logs
great, We have done quite a lot so far, now lets see how it applies with a case study
Good Launch
Bad launchAdd fail|exception|error to search strings
DDC not AuthorizedPolicy found that was needed but not enabledResolution to explicitly enable access
Citrix Scout used to collect environment information and CDF tracesXenDesktop Collector the replacement for Scout with the next Feature Pack release of XenDesktop
Desktop Director is a tool that will utilize role-based permission sets to support the daily usage of Citrix products. It enables support teams to perform basic maintenance tasks and to monitor and troubleshoot system issues. Desktop Director 1.0 wasintroduced with XenDesktop5 and Desktop Director 2.0 supports troubleshooting XenAppsessions.Role-based access control – assign appropriate permissions to specific roles to perform certain operation. Full administrator can view all and make changes. Read-only administrator can view all but cannot perform tasks. Help desk administrator can performday-to-day monitoring and maintenance tasks (restarting desktops or logoff sessions).
At Citrix Services - we’re Citrix consultants, teachers and support engineers and we’re all about one thing: making sure you succeed.With our help, you’ll deploy high-performance, robust virtualization and networking projects, faster – with dramatically lower risk and higher return.The best Citrix architects and administrators are the ones who never stop learning – and Citrix Education is here to help you learn those skills.Citrix Consulting gives you direct access to our most experienced virtualization and networking experts.When it’s complex; when it’s mission-critical; when it’s big; That’s when Citrix consultants can really help.On your virtualization journey, you’ll want always-on support from people who really care about your success.There’s no better insurance for your Citrix investment than with Citrix Support.
Secrets of the Citrix Support Ninjas is a FREE eBook available next week.The eBook contains 40 insider troubleshooting tips for administrators.So the purpose of the eBook is to help administrators like you keep your Citrix deployments on track.We’ve collected some of their best tips and tricks for running robust Citrix environments and packaged them up into a free eBook.In it, you’ll discover some of the little-known tricks that our own support people use every day to tune, tweak, troubleshoot and test Citrix solutions. You may know a few of these tips. But you probably don’t know them all.And – you never know – you might discover just one that will change your life as an administrator.Let me give you a sneak peak now.