Fix Citrix NetScaler HTTP1.1 Internal Server Error 43550 – Troubleshooting & Solutions

Recently, I encountered an issue with Citrix ADC/Netscaler NS13.1.51.15.nc. When accessing Citrix desktops through ADC, after entering the account password for authentication, the following error appeared:

Http/1.1 Internal Server Error 43531

As shown in the figure below:

There have been no recent changes to Citrix or the network. I checked the StoreFront certificate and the ADC certificate, and neither had expired. I also verified that the URL entered for the gateway in SF matched the ADC AG URL and that the configurations were all correct.

Understanding Citrix NetScaler HTTP/1.1 Internal Server Error 43550

The HTTP/1.1 Internal Server Error 43550 in Citrix NetScaler (also known as Citrix ADC) indicates a backend communication or configuration failure. This usually happens when:

• Backend servers are unreachable or misconfigured

• SSL certificate bindings are incorrect

• Gateway or load balancing settings are mismatched

• The appliance is overloaded

After troubleshooting without success, I sought assistance. The engineer suggested trying to modify the Citrix Gateway Session Profile in the Web Interface Address's Change FQDN to StoreFront's IP address (if Load Balancing is configured, change it to the StoreFront VIP). 

The reason seems to be that the LB address can't be resolved via DNS, need to check if the ADC is pointing to an IP or DNS.

The steps to fix HTTP/1.1 Internal Server Error 43550

1. Log in to the ADC/Netscaler web console

2. Navigate to Citrix Gateway → Policies → Session → Session Profiles

TIP: AC_OS refers to the client access session profile, and AC_WB refers to the session profile accessed through the browser WEB client visited session profile

3. Click to enter the AC_WB profile, then click "Published Applications", change the FQDN in "Web Interface Address" to the StoreFront IP address (or the SF LB VIP address), and keep other configurations unchanged. The configuration is shown in the figure below:

4. Click "OK" to save, and it will take effect immediately.

5. If users also need to access desktops through the client, perform the same operation on the AC_OS session profile using the method described above.

At this point, users can successfully access desktops and applications through Citrix ADC/Netscaler.

Best Practices to Prevent Error 43550

• Always test backend server connectivity after configuration changes.

• Keep NetScaler firmware updated to avoid known bugs.

• Enable logging and monitoring to detect issues early.

• Maintain proper SSL/TLS configurations for security and stability.

Fix VMware ESXi Cannot Expand VMFS Datastore – Step-by-Step Troubleshooting Guide

 In a user's VMware production environment, ESXi storage inevitably encounters situations where expansion is needed. At this point, the RAID needs to be expanded at the storage underlying layer, and then the datastore needs to be expanded in the ESXi Host Client. Some users may encounter errors during expansion, as shown below:

The RAID has been expanded at the storage underlying layer, and the ESXi device list also shows that the storage capacity has increased. Still, it has not yet been allocated to the datastore for virtual machine use. At this time, an error occurs when expanding the datastore in the web interface: “Cannot expand VMFS datastore XXX - Cannot update disk partition for /vmfs/devices/disks/naa.xxxxxx ”As shown in the figure below:

Understanding the “Cannot Expand VMFS Datastore” Error

In VMware ESXi, a VMFS datastore stores virtual machine files. Sometimes, when you try to expand the datastore after adding new storage capacity, you may encounter the error:

“Cannot expand VMFS volume: capacity available for expansion is not enough.”

Common causes include:

• Disk partitions are not updated after adding new LUNs.

• Unrecognized free space due to partition alignment issues.

• Storage device firmware incompatibility.

Solutions to fix VMFS datastore that can not expand

I. This error may be caused by a bug in the ESXi Host Client (ESXi's web portal). You can prioritize trying to expand in vCenter Server; the operation method is the same as in the ESXi Host Client.

II. Use the command line for the expansion operation

First, it is recommended to back up virtual machines in the datastore or migrate them to other hosts before the operation to avoid data loss!!!

The operation method is as follows:

1. Enable SSH service in ESXi (disabled by default, and it is recommended to keep it disabled normally)

2. Use an SSH tool such as MobaXterm, Putty, etc., connect and log in to ESXi with the root account.

3. Use the following command to view disk partitions. The naa number of the disk can be viewed by logging into the web interface and checking the device path (replace the naa number with the actual naa number in your environment)

  
partedUtil getptbl "/vmfs/devices/disks/naa.6c81f660ce9d9e0027582d5d2b3b3340"

The output is roughly as shown below (there may be errors in the output, just ignore them)

4. Use the following command to view the end sector of the VMFS datastore (replace the naa number with the actual naa number in your environment)

  
partedUtil getUsableSectors "/vmfs/devices/disks/naa.6c81f660ce9d9e0027582d5d2b3b3340"

From the command output, the end sector is 14058258398

5. Then use the following command to expand the capacity

  
partedUtil resize "/vmfs/devices/disks/naa.6c81f660ce9d9e0027582d5d2b3b3340" 1 2048 14058258398

“1 2048 14058258398” in the above command refers to “partition number, starting sector, ending sector” respectively. Please modify it according to the actual environment.

6. There may be an error after expansion, and continue to run the following command

  
vmkfstools --growfs "/vmfs/devices/disks/naa.6c81f660ce9d9e0027582d5d2b3b3340:1" "/vmfs/devices/disks/naa.6c81f660ce9d9e0027582d5d2b3b3340:1"

The “:1” refers to the partition number of the partition to be expanded, which can be viewed in the web or in the command output of the third step. For example, if my partition number is 1, it is “:1”, as shown in the figure below.

7. After completing the above operations, you can use the "df -h" command to check whether the storage expansion is complete, or return to the web page to view the storage capacity.

Best Practices to Prevent Expansion Errors

• Always take a datastore backup before expansion.

• Keep ESXi host firmware and storage array firmware updated.

• Use the same block size across LUNs to avoid format conflicts.

Final note:

If you find that the storage still cannot be expanded after following these steps, here's another possible solution (I haven't encountered this myself, but other users online have reported it). This error might also occur if the newly added storage wasn't properly formatted or if there are residual data partitions that weren't cleaned up. In such cases, you may need to reset the partition table or reformat the data storage. Since this involves storage operations, it's strongly recommended to back up your virtual machine and data before performing any operations!!!

VMware ESXi Virtual Disk Type Conversion – Thin, Thick, and Eager Zeroed Guide

 In a VMware virtualization environment, the three ESXi disk types mentioned in previous articles (such as Thick Provision Lazy Zeroed, Thick Provision Eager Zeroed, and Thin Provision) not only affect the virtual machine's Detailed Explanation of VMware ESXi Disk Provisioning Typesperformance, space consumption, and creation speedPerformance, space occupation, and creation speed, but also directly influencestorage resource planning and post-maintenance strategies.

However, in real-world operations, many administrators encounter these tricky situations:

• Initially selected the wrong disk type—When deploying, thin provisioning was used to save space, but in the production environment, write volumes were large, and performance fluctuations were obvious; or thick provisioning eager zeroed was used, which was slow to create and fully occupied storage, leading to insufficient space for new projects.

• Changes in business scenarios—Thin provisioned disks in the test environment need to be converted to thick provisioned for production; after enabling VMware Fault Tolerance or Multi-Writer features in the cluster, disks must be thick provisioned eager zeroed.

• Storage over-provisioning risk—Thin-provisioned disks suddenly run out of physical space during peak write times, causing virtual machine downtime.

• Pitfalls during expansion or migration—Online expansion of thick provisioned eager zeroed disks results in newly allocated space becoming thick provisioned lazy zeroed, affecting shared disk functionality.

The commonality of these issues is that all require disk type conversion. Below is a detailed analysis of how to convert disk types, including converting from thin provisioning to thick provisioning, or vice versa.

The operating steps are as follows:

I. Converting Thin Provision to Thick Provision

1. Confirm that the virtual disk type is currently Thin Provision

2. This virtual machine uses a thin-provisioned disk type, with 50 GB of storage allocated, but only 1.86 GB of storage space is actually occupied.

3. Click "Edit Settings"

4. Confirm which storage the current virtual machine files are in, and click the storage location name to enter the storage page

5. Find the virtual machine folder in the browse storage, and find the vmdk file to be converted, then click "Expand" above (if the expand option is grayed out, it means the current disk storage format is not Thin Provisioning or the virtual machine is not in a powered off state)

6. Confirm expansion

7. Wait for the expansion task to complete

8. Go back to the virtual machine summary page, the hard disk allocation is still 50 GB, but it has become Thick Provision Eager Zeroed, and the storage space occupied is 50 GB (occupies as much as allocated)

9. Open the SSH service of ESXi (the previous ESXi upgrade article, VMware ESXi complete upgrade steps illustrated how to open the SSH service), use the following command to query the virtual machine's vmid, where vm-name is replaced with the virtual machine's name

  
vim-cmd vmsvc/getallvms|grep -i vm-name

10. Copy and record the VMID obtained in the previous step, which is 523 here, then run the following command to reload the virtual machine's VMX file, replacing <VMid> with the actual VMid

  
vim-cmd vmsvc/reload VMid

II. Convert thick provision to thin provision

There are two ways to convert a thick provisioned disk to a thin provisioned disk: one is to use migration when there are multiple storage paths in the environment, and the other is to use cloning when there is only one storage path in the environment. Of course, cloning can also be used for conversion when there are multiple storage paths.

The following describes the operating steps for using both methods:

A. Use the migration function for conversion (when there are multiple storage paths)

1. Right-click the virtual machine name and click "Migrate"

2. Select "Change storage only" or "Change compute resource and storage"

3. Select the target storage path, and select "Thin Provision" as the virtual disk format.

(If "Configure per disk" is enabled, you can set different storage locations and disk formats for different disks)

4. Confirm the migration information and click Finish

5. After the migration is complete, return to the virtual machine summary page to see that the disk has been converted to thin provision, and the storage space occupied has also shrunk, becoming consumed as much as used.

B. Use the cloning function for conversion (when there is a single storage path)

1. Right-click the virtual machine and click "Clone" → "Clone to Virtual Machine"

2. Enter the virtual machine name and select the location

3. Select Compute Resource

4. Select the storage location and choose the virtual disk format as Thin Provision(Similar to migration, you can enable the per-disk configuration feature)

5. Select cloning options as needed

6. Confirm cloning information and click Finish

7. After the cloning is complete, return to the virtual machine summary page and confirm that the disk has been switched to thin provisioning, and the storage space has also been reduced, becoming 'use as much as occupied'.

The virtual disk format conversion is now complete!

Citrix NetScaler Password Recovery – Step-by-Step Admin Account Reset Guide

 1. Foreword

Recently, after upgrading Citrix Netscaler for a client, I found that I could no longer log in with the original password. After checking, it was found that the password saving format had changed after the previous version 10.5 was upgraded to 13.1. This caused the original password to be unusable for login, so the nsroot password needed to be reset to log in.

Note: This operation method is only for Citrix ADC hardware devices.

Understanding Citrix NetScaler Password Recovery

The Citrix NetScaler (also known as Citrix ADC) is a critical networking appliance used for load balancing, application delivery, and secure remote access. If you forget the admin password, you may lose access to key configurations, affecting your entire network infrastructure.

Fortunately, Citrix provides a secure recovery process to reset the administrator password without losing system settings.

When You Might Need a Password Reset

• Forgotten or misplaced admin credentials

• Security policy enforcement requiring account resets

• Compromised accounts after suspicious login activity

• Inherited systems from previous IT staff without password documentation

2. Preparations

1. Preparation: Serial adapter, serial (male) to network cable, serial (female) to USB.

2. Connect as shown in the figure, and connect to Netscaler with SSH.

3. Restart Netscaler

3. Step-by-Step Citrix NetScaler Password Recovery Process

1. When you see the interface shown in the figure from tools like Putty, click Ctrl+C, then enter boot -s.

2. Enter the following interface and start the operation.

3. /sbin/fsck/dev/ad0s1a and Enter

4. Then go with the following codes:

  
 /sbin/mount /dev/ad0s1a /flash
 cd /flash/nsconfig
 df
 grep -v “set system user nsroot ns.conf” > new.conf
 mv ns.conf old.ns.conf
 mv new.conf ns.conf

5. After restarting, you can log in with the default password.

Best Practices for Citrix NetScaler Password Security

• Use strong, complex passwords with at least 12 characters.

• Enable multi-factor authentication for admin accounts.

• Store credentials securely in an encrypted password vault.

• Regularly rotate passwords to prevent brute force attacks.

 Related Security Hardening Tips

• Restrict management access to specific IP ranges.

• Apply the latest Citrix firmware updates.

• Perform regular configuration backups for quick recovery.

Fix Citrix ADC Gateway Remote Code Execution Vulnerability – Step-by-Step Security Patch Guide

Fix Citrix ADC Gateway Remote Code Execution Vulnerability 

 I. Foreword

A risk advisory from Citrix regarding NetScaler ADC and NetScaler Gateway, with vulnerability ID CVE-2023-3519.Vulnerability level: Critical, Vulnerability score: 9.8

Vulnerability Impact: Hackers can use this vulnerability to bypass any authentication and directly access the shell on NetScaler devices configured with Gateway (VPN virtual server, ICA proxy, CVPN, RDP proxy) or AAA virtual server, then penetrate the internal network to perform illegal operations (Test environment has been verified).

Understanding the Citrix ADC Gateway RCE Vulnerability

Citrix ADC (formerly NetScaler) Gateway is widely used for secure remote access to corporate networks. Recently, a Remote Code Execution (RCE) vulnerability was discovered, allowing attackers to execute arbitrary commands on unpatched appliances. This vulnerability is often referenced in Citrix Security Bulletins and tracked under a CVE (Common Vulnerabilities and Exposures) ID.

If exploited, attackers can:

• Gain unauthorized administrative access.

• Install malware or backdoors.

• Exfiltrate sensitive corporate data.

• Disrupt business-critical services.

Check If Your Citrix ADC is Vulnerable

1. Log in to the Citrix ADC CLI or web UI.

2. Check your software build number using:

     
   show version
   

3. Compare against Citrix’s latest security advisory.

II. Experimental Results

The figure shows that by using Kali, bypassing the NetScaler management interface, and entering shell mode. The image below is a screenshot of the test environment verification, and the verification steps have been omitted.

III. Affected Versions (CVE-2023-3519)

You can check if your current version is within the affected range based on the list below.

Component	Affected Version	Secure Version
Citrix: NetScaler ADC, NetScaler Gateway	13.0 < 13.0-91.13	13.0 >= 13.0-91.13
Citrix: NetScaler ADC, NetScaler Gateway	13.1 < 13.1-49.13	13.1 >= 13.1-49.13
Citrix: NetScaler ADC	12.1-FIPS < 12.1-55.297	12.1-FIPS >= 12.1-55.297
Citrix: NetScaler ADC	12.1-NDcPP < 12.1-55.297	12.1-NDcPP >= 12.1-55.297
Citrix: NetScaler ADC	13.1-FIPS < 13.1-37.159	13.1-FIPS >= 13.1-37.159

IV. Preparations before upgrading

1. We will mainly demonstrate upgrading Citrix ADC 8910 (hardware). The current version is Release: NS13.1 33.47.nc. 

Upgrade tool: MobaXterm (SSH login to execute commands and upload files)

2. Back up the configuration file, open the location shown in the figure: System->Backup and Restore

3. Enter the backup name, as shown in Figure 

4. Select to download the file and save it locally for easy recovery.

V. Command line upgrade steps

1. Log in to SSH, save the configuration, and save nsconfig

2. Upload the upgrade package to the directory var/nsinstall. The version is: build-13.1-49.13_nc_64.

3. Unzip the file, tar -zxvf build-13.1-49.13_nc_64.tgz

4. Click Install

5. Restart

6. Upgrade completed

7. After the upgrade is completed, to prevent occupying space, the installation package needs to be deleted

V. GUI interface upgrade steps

1. Log in to the Citrix ADC Web page

2. Open the update path

3. Select the upgrade package

4. Click Upgrade

5. Restart after the upgrade is completed.

By following this guide, you protect your Citrix ADC Gateway from one of the most dangerous threats in recent years — a remote code execution attack that could compromise your entire network.