netscaler_threathunter.sh: a bash2 compatible script to digital forensic and incident response on Citrix ADC

Hello everyone, this is Gabriele Pippi, from the Certego Purple Team.

Today we are ready to publish the tool we are using to perform digital forensic and incident response on Citrix Netscaler ADC based on FreeBSD.

Our point of view on CVE-2019-19781 exploitation

After the first mad release of the POC for CVE-2019-19781 by the github account @projectzeroindia and first exploitations, trustedsec and other members of the community provided the first details to carry out digital forensic and incident response operations.

Our Incident Response Team from the first engagements has verified the following conditions:

  1. All the Netscaler ADC analyzed run on FreeBSD kernel version. Unlike in FreeBSD core, there were several utilities installed on the machine, such bash2, used for this script.
  2. The code executed remotely with the CVE ran with nobody user, there were no easy privilege escalations and the persistence on the machine was easily circumscribed; we found cronjobs, php webshells and perl scripts.
  3. Some attackers tried to exfiltrate a file called ns.conf, which contains a large number of sensitive information, especially encrypted passwords and hashes. This is an advanced way to exfiltrate ns.conf.

After the first engagements there were some important implications:

  1. The hashes contained in the ns.conf file were already supported and crackable by hashcat since 2014Reference
  2. Some passwords in the ns.conf are encrypted with AES256-CBC or AES256-ECB algorithms using a well-known key, compiled in a library. These passwords can be decrypted immediately. Among these we also found AD domain admin. https://dozer.nz/citrix-decrypt
  3. The default root password of Citrix Netscaler AMIs is set to the instance ID: this makes trivial a privilege escalation on all installations of this type. Reference
  4. Recently, some actors are starting to exploit this CVE for ransomware campaigns. Reference

Certego Netscaler Threathunter

I developed this bash2 compatible script in order to automate all checks and facilitate threat hunting operations.
The script is designed to provide output to an analyst who will then carry out the analysis.

Note #1: we take no responsibility for any improper use of this script. We recommend using it with caution on critical systems in production.

Note #2: except for its optional log file, the script does not perform any writing operations, does not need any installation and can also be launched in fileless mode.

Requirements:

  • Netscaler running on FreeBSD Kernel Version.
  • nsroot or root privileges.

Purposes:

This script aims to:

  • validate patch of CVE-2019-19781.
  • enumerate persistences and artifacts related to CVE-2019-19781.
  • guide users to change ns.conf credentials to avoid future campaigns and to patch quickly.
  • spot possible advanced privilege escalation.

For further technical details and to download the script, please refer to the official github project Netscaler_Theathunter.

All Citrix Official Advisories

About the author

Gabriele Pippi, Threat Research Lead Engineer (Twitter)