# Troubleshooting ## Verifying discovery tool connectivity to vCenter When you experience VMware module configuration errors follow these steps to verify connectivity: **Access the discovery tool VM** + Log-in to the discovery tool VM, open Remote Console in vCenter + Username: discovery + Password: password **Test vCenter Connectivity** 1. Test vCenter API Access: ``` curl -v --insecure -u : https://:443/mob ``` 1. Expected Success Output: ``` [ec2-user@discoverytool ~]$ curl -v --insecure -u : https://vcsa/mob > tmp.txt % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 192.168.2.125:443... * Connected to vcsa (192.168.2.125) port 443 (#0) ... * Connection #0 to host vcsa left intact ``` **Test SSL Certificate** 1. Run this command: ``` openssl s_client -showcerts -servername -connect :443 ``` 1. Expected Success Output: + Should show vSphere certificate details + Verifies SSL/TLS connectivity on port 443 ``` [ec2-user@discoverytool ~]$ openssl s_client -showcerts -servername vcsa -connect vcsa:443 CONNECTED(00000003) depth=0 CN = vcsa.onpremsim.env, C = US verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 CN = vcsa.onpremsim.env, C = US verify error:num=21:unable to verify the first certificate verify return:1 --- Certificate chain 0 s:/CN=vcsa.onpremsim.env/C=US i:/CN=CA/DC=vsphere/DC=local/C=US/ST=California/O=vcsa.onpremsim.env/OU=VMware Engineering -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- --- Server certificate subject=/CN=vcsa.onpremsim.env/C=US issuer=/CN=CA/DC=vsphere/DC=local/C=US/ST=California/O=vcsa.onpremsim.env/OU=VMware Engineering --- ``` ## WinRM Troubleshooting If you're experiencing connectivity issues with WinRM, follow these steps to test the connection. These steps also apply to Hyper-V connectivity issues, because the discovery tool uses WinRM to communicate with Hyper-V hosts. Test basic WinRM connectivity using ports 5985 (HTTP) and 5986 (HTTPS). We need to make sure that connectivity works on port 5986 (HTTPS) ``` # Check WinRM listener configuration winrm enumerate winrm/config/listener # Note: Replace with the target computer's hostname or IP address. Adjust the username and password as needed. # Test WinRM connection on port 5985 (HTTP) $cred = Get-Credential Test-WSMan -Computer -Authentication Negotiate -Credential $cred -Port 5985 # Test WinRM connection on port 5986 (HTTPS) Test-WSMan -Computer -Authentication Negotiate -Credential $cred -Port 5986 ``` If the above tests fail, try establishing a PowerShell session with certificate validation disabled: ``` $cred = Get-Credential $so = New-PsSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck Enter-PSSession -ComputerName -Credential $cred -Port 5985 -SessionOption $so ``` ## Kerberos troubleshooting If you experience Kerberos authentication failures when collecting data from Windows servers, use the following sections to diagnose and resolve common issues. ### Verify network requirements Before you troubleshoot Kerberos authentication, verify that the discovery tool can reach the required network endpoints. **To verify network requirements for Kerberos** 1. Verify DNS resolution to the domain controller. Run the following command from the discovery tool VM: ``` nslookup dc01.example.com ``` Alternatively, you can use `dig`: ``` dig dc01.example.com ``` If DNS resolution fails, verify that the discovery tool VM is configured to use a DNS server that can resolve your Active Directory domain. Check `/etc/resolv.conf` and confirm that the nameserver entries point to your domain DNS servers. 1. Verify connectivity to the Key Distribution Center (KDC) on port 88. Run the following command: ``` nc -zv dc01.example.com 88 ``` Expected output: ``` Connection to dc01.example.com 88 port [tcp/kerberos] succeeded! ``` If the connection fails, verify that no firewall rules block traffic from the discovery tool VM to the domain controller on port 88. 1. Verify connectivity to the target Windows servers on WinRM ports. Run the following commands: ``` nc -zv 5985 nc -zv 5986 ``` If the connection fails, verify that WinRM is enabled on the target server and that firewall rules allow inbound traffic on ports 5985 and 5986. ### Common Kerberos issues The following are common Kerberos issues and their solutions. **Case sensitivity errors** Symptom: You receive a "Server not found in Kerberos database" error during authentication. This error typically occurs when the Kerberos realm name is not in uppercase. Kerberos realms must be specified in uppercase in your `krb5.conf` file. For example, use `EXAMPLE.COM` instead of `example.com`. **Account lockout prevention** The discovery tool uses a backoff mechanism to prevent account lockouts from repeated failed authentication attempts. If your service account becomes locked out, you can reset the collection process by stopping and then starting the collection module through the discovery tool web UI at `https://:5000`. **kinit fails from CLI** The following table lists common `kinit` errors and their solutions. | Error | Cause | Solution | | --- | --- | --- | | Cannot find KDC for realm | The KDC hostname or IP address is not reachable, or the realm is not configured in krb5.conf. | Verify that the krb5.conf file contains the correct KDC hostname and realm. Confirm DNS resolution and network connectivity to the KDC on port 88. | | Preauthentication failed | The password for the service account is incorrect. | Verify the password and try again. If the account is locked out, unlock it in Active Directory before you retry. | | Client not found in Kerberos database | The principal name does not match any account in Active Directory. | Verify that the principal name matches the account name exactly, including case. Use the format username@REALM with the realm in uppercase. | | Cannot resolve network address for KDC | DNS cannot resolve the KDC hostname. | Verify DNS configuration in /etc/resolv.conf. Confirm that the DNS server can resolve the KDC hostname. Test with nslookup or dig. | **Collection fails despite successful kinit** If `kinit` succeeds but data collection still fails, check the following: 1. Verify that the principal name used for collection matches the case used during `kinit` exactly. 1. Verify that the service account has the required permissions on the target servers. 1. Verify that WinRM is enabled on the target servers. 1. Verify that the hostname used for collection matches the hostname registered in Active Directory. **Kerberos works for some servers but not others** If Kerberos authentication succeeds for some servers but fails for others, investigate the following areas: If your servers span multiple Active Directory domains, configure a separate Kerberos credential for each domain. Make sure that your `/etc/krb5.conf` file includes entries for all realms. Each domain requires its own credential with the correct `username@REALM` principal. Compare the WinRM configuration on a working server with a failing server. Run the following command on each server: ``` winrm get winrm/config ``` Test connectivity with Remote Desktop to isolate the issue. The discovery tool uses the format `username@DOMAIN`, while Remote Desktop uses the format `DOMAIN\username`. Verify that the service account is a member of the local Administrators group on the failing server. Run the following command on the target server: ``` net localgroup Administrators ``` WMI requires local administrator privileges to access operating system information. SQL Server collection also requires that the service account has local administrator access on the target server. ### Kerberos configuration checklist Use the following checklist to verify your Kerberos configuration before you start data collection. + The `krb5.conf` file exists on the discovery tool VM. + The realm name is in uppercase in `krb5.conf`. + Running `kinit` with the service account succeeds without errors. + Running `klist` shows a valid, non-expired ticket. + The principal name matches the Active Directory account name exactly. + DNS resolution works for the KDC hostname. + Network connectivity to the KDC on port 88 is confirmed. + Network connectivity to target Windows servers on ports 5985 and 5986 is confirmed. + (Multi-domain) Each Active Directory domain has its own credential configured in the discovery tool, and `krb5.conf` contains `[realms]` and `[domain_realm]` entries for all domains. ## Oracle Database troubleshooting To diagnose Oracle Database collection issues, such as missing data or errors, check the following. **Connection refused or timeout** Symptom: Oracle collection status shows a connection error for a server. To troubleshoot this issue, check the following: + Verify that the Oracle listener is running on the target host: `lsnrctl status` + Verify network connectivity from the discovery tool to the Oracle host on port 1521 (or your custom port): `nc -zv 1521` + Verify that firewall rules allow inbound connections on the Oracle listener port. + Verify the service name by running `lsnrctl services` on the Oracle host. If the service name is incorrect, the Oracle listener rejects the connection. **Authentication failures (ORA-01017)** Symptom: Collection fails with an invalid username or password error. To troubleshoot this issue, check the following: + Verify that the Oracle service account exists and is not locked: `SELECT account_status FROM dba_users WHERE username = 'DISCOVERY_USER';` + Verify the password is correct by connecting manually: `sqlplus discovery_user/@:1521/` + If the account is locked, unlock it: `ALTER USER discovery_user ACCOUNT UNLOCK;` **Insufficient privileges (ORA-01031)** Symptom: Connection succeeds but collection returns incomplete data. To troubleshoot this issue, check the following: + Verify that SELECT\_CATALOG\_ROLE is granted: `SELECT * FROM dba_role_privs WHERE grantee = 'DISCOVERY_USER';` + Grant the required role if missing: `GRANT SELECT_CATALOG_ROLE TO discovery_user;` **Manual credential shows error but auto-connect works** When you pin a credential manually to a server, the discovery tool does not fall back if the connection fails. Verify that the port and service name that you configured on the credential match the Oracle listener on that specific server. If the server has a non-standard port or service name, update the credential configuration accordingly. **OS-level fallback not detecting Oracle** If no database credentials are configured and the OS-level fallback does not detect Oracle: + Verify that SSH or WinRM OS credentials are configured and working for the server (check OS metrics collection status). + For Linux hosts, verify that `/etc/oratab` exists or that Oracle process monitor (`pmon`) processes are running. + For Windows hosts, verify that Oracle registry entries exist under `HKLM\SOFTWARE\Oracle` or that `oracle.exe` processes are running. ## SNMP Troubleshooting **Access the discovery tool VM** + Log-in to the discovery tool VM, open Remote Console in vCenter + Username: discovery + Password: password **Install SNMP Tools (if needed)** + `sudo yum install net-snmp-utils -y` **Test SNMP Connection to Linux Servers** 1. `snmptable -v 2c -c .1.3.6.1.2.1.6.13.1` 1. Example: ``` #SNMPv2c: snmptable -v 2c -c public 192.168.1.100 .1.3.6.1.2.1.6.13.1 #SNMPv3 (with authentication): snmptable -v 3 -u -a MD5 -A 192.168.1.100 .1.3.6.1.2.1.6.13.1 #SNMPv3 (with privacy): snmptable -v 3 -u -a MD5 -A -x DES -X 192.168.1.100 .1.3.6.1.2.1.6.13.1 ``` ## Network collection errors ### a terminal is required to read the password **Error:** ``` ss command failed on : sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper sudo: a password is required ``` The ss command is prompting for user password. The configured ssh user must be in the sudoers group and be configured with passwordless sudo for the ss/netstat command. To configure passwordless sudo: 1. Create a new sudoers file: ``` sudo vi -f /etc/sudoers.d/ ``` 1. Add the line: ``` ALL=(ALL) NOPASSWD: /usr/sbin/ss, /usr/bin/netstat ``` 1. After this change, running `sudo ss -tnap` and `sudo netstat -tnap` should execute without prompting for a password ### Network collection ran without sudo If you see the following warning on the **Discovered inventory** page: ``` Network collection ran without sudo. Process-level connection data may be missing. ``` This warning indicates that the SSH user account does not have sudo access on the target server. Without sudo, the discovery tool can still collect network connection data, but it cannot determine which process owns each connection. To collect complete process-level connection data, ensure the SSH user has sudo access on the target server. ## OS metrics collection errors ### Missing server UUID for Linux servers If the discovery tool is unable to collect the server UUID (showing as empty or missing) for Linux servers, verify that the SSH credentials configured for those servers have sudo privileges. The tool uses `dmidecode` to read the server UUID. If `dmidecode` is not installed, the tool falls back to reading `/sys/class/dmi/id/product_uuid`, which also requires sudo access. Without sudo, neither method can retrieve the UUID. **Resolution:** Ensure the SSH user account provided to the discovery tool has sudo access on the target Linux servers. ## Access issues in Discovered inventory If you see a message in **Server collection status** such as Missing credentials, or Access denied: 1. Select the server on the table of discovered servers. 1. Choose **Manage access credential** You can choose to: 1. Select alternative credentials from the **Select credentials** dropdown. 1. Select **Use new credentials** and provide new credentials. 1. **Save**. The discovery tool retries the connection after you save your changes. ## SSH key authentication troubleshooting **Testing SSH key connectivity from the discovery tool** If SSH key authentication fails, verify connectivity from the discovery tool to the target server: 1. Log in to the discovery tool (via vSphere console or SSH to the Linux host). 1. Test SSH connectivity using your private key: ``` ssh -i /path/to/private_key -o StrictHostKeyChecking=no @ ``` 1. If the connection succeeds, the issue is with how the key was uploaded to the discovery tool. Re-upload the key and verify the username matches. 1. If the connection fails, check the error message in the following table. | Error message | Cause | Resolution | | --- | --- | --- | | Permission denied (publickey) | The public key is not in the target server's authorized\_keys file, or the username is wrong. | Add the public key to \~/.ssh/authorized\_keys on the target server for the correct user. Verify file permissions: chmod 700 \~/.ssh && chmod 600 \~/.ssh/authorized\_keys. | | Connection timed out after 20s | Port 22 is not reachable from the discovery tool. | Verify that port 22 is open between the discovery tool and the target server. Check firewalls and security groups. | | Connection refused | SSH service is not running on the target server. | Start the SSH service: sudo systemctl start sshd. | | Invalid SSH key for credential '{{name}}' | The key format is not supported, the key data is corrupted, or the passphrase is missing or incorrect. | Verify the key format is RSA, ECDSA, or Ed25519 in PEM, OpenSSH, or PKCS\#8 format. If the key is encrypted, make sure the passphrase is correct. | ## Linux installer troubleshooting **Port 5000 is already in use** **Symptom:** The discovery tool service fails to start after installation. **Resolution:** Identify and stop the process using port 5000: ``` sudo ss -tlnp | grep :5000 ``` Stop the conflicting process, then restart the discovery tool: ``` sudo ./AWS-Transform-discovery-tool.sh start ``` ## Common error messages This table describes common error messages and their explanations: | Message | Location | Explanation | | --- | --- | --- | | A password has already been created | Create password page | Race condition when two users create passwords at the same time; refresh | | Export failed | Inventory page | Retry or send logs | | An on-demand collection is already in progress | Inventory page | Race condition when two users start manual collections at the same time; try again after the current manual collection is finished | | Command timed out after 60s | Server collection status | A command on the target server did not complete within 60 seconds. This can occur on heavily loaded servers. Retry the collection or investigate the target server's load. | | One or more credentials contain unknown UUIDs | OS access page | Race condition when two users edit OS credentials at the same time; try again | | Invalid password | Sign-in page | Incorrect password for logging in; contact admin or reach out | | Your session has expired. Please log in again. | Sign-in page | Session has timed out, need to login again | | An internal error occurred | Various pages | Retry or send logs |