SingHealth AddOn

Question 1

From 26 June 2018, the attacker began querying the database from
Citrix Server 2 using the A.A. account.
3 types of “SQL” queries which the attacker ran:

Answer 1

• (i) reconnaissance on the schema of the SCM database,
• (ii) direct queries relating to particular individuals, and
• (iii) bulk queries on patients in general.

Question 2

The attacker was able to retrieve the following information from the
SQL queries:

Answer 2

1. The Prime Minister’s personal and outpatient medication data;
2. The demographic records of 1,495,364 patients, including their
   names, NRIC numbers, addresses, gender, race, and dates of birth;
3. The outpatient dispensed medication records of about 159,000 of
   the 1,495,364 patients mentioned in sub-paragraph (b) above

Question 3

• The copying and exfiltration of data from the SCM database was
stopped on 4 July 2018, after staff from IHiS discovered ____

Answer 3

d the unusual
queries and took steps to prevent any similar queries from being run
against the SCM database.

Question 4

Although no data queries to the SCM database or exfiltration of patient
records were detected after 4 July 2018, there was malicious activity in
the SingHealth network on 18 and 19 July 2018, which suggested that:

Answer 4

• the attacker was trying to establish a fresh pathway into the network;
and
• the attacker had established multiple footholds in the network and
had re-entered the network through one of these hitherto unknown
footholds

Question 5

On 18 July 2018, phishing emails were sent to a number of recipients in
various SingHealth institutions.

Answer 5

• One of the recipients of the email was the user of a previously
infected workstation – the PHI 1 Workstation.
• The email contained content similar to the earlier mentioned publicly
available hacking tool, and would run automatically when the mail
was previewed or read.
• It was also configured to lead to callbacks to a C2 (command&control)
server.

Question 6

• After detection of malware on and communications from the S.P.
server

Answer 6

CSA recommended that internet surfing separation should be
implemented, to prevent the attacker from exercising command and
control over any remaining footholds it may have in the network.
• Internet surfing separation was implemented on 20 July 2018.
• No further signs of malicious activity were detected thereafter

Question 7

CONTRIBUTING FACTORS

LEADING TO THE CYBER ATTACK

Answer 7

Network connections between the SGH Citrix
servers & SCM database were allowed
Lack of monitoring at the SCM database for
unusual queries and access
SGH Citrix servers were not adequately
secured against unauthorised access
Weak controls over and inadequate
monitoring of local administrator accounts
Lack of sight over and mismanagement of the
S.A. service account
Internet connectivity in the SingHealth IT
network increased the attack surface
Versions of Outlook used by IHiS were not patched
against a publicly available hacking tool
Coding vulnerability in the SCM application

Question 8

1

Answer 8

Network connections between the SGH Citrix

servers & SCM database were allowed

Question 9

2

Answer 9

Lack of monitoring at the SCM database for

unusual queries and access

Question 10

3

Answer 10

SGH Citrix servers were not adequately

secured against unauthorised access

Question 11

4

Answer 11

Weak controls over and inadequate

monitoring of local administrator accounts

Question 12

5

Answer 12

Lack of sight over and mismanagement of the

S.A. service account

Question 13

6

Answer 13

Internet connectivity in the SingHealth IT

network increased the attack surface

Question 14

7

Answer 14

Versions of Outlook used by IHiS were not patched

against a publicly available hacking tool

Question 15

8

Answer 15

Coding vulnerability in the SCM application

Question 16

r1

Answer 16

enhanced security structure and readiness must be adopted by IHIS and Public Health Institutions

Question 17

r2

Answer 17

the cyber stack must be reviewed to assess if it is adequate to defend and respond to advanced threats

Question 18

r3

Answer 18

staff awareness on cybersecurity must be impvoed, to ehance capacity to prevent, detect and respond to security incidents

Question 19

r4

Answer 19

ehanced security checks must be performed especailly on CII systems

Question 20

r5

Answer 20

Priviledged admin accounts must be subject to tighter control and greater monitoring

Question 21

r6

Answer 21

incident response process must be improved for more effective response to cyber attacks

Question 22

r7

Answer 22

partnerships between industry and government to achieve a higher level of collective security

Question 23

Network connections between the SGH Citrix
servers & SCM database were allowed
1

Answer 23

• The network connection was a critical pathway to the SCM database,
over which the attacker was able to make SQL queries to and retrieve
data from the SCM database.
• but for this open network connection, the SCM database was
adequately protected within the H-Cloud perimeter defences, and the
attacker would not have been able to access the SCM database as
easily.
• This open connection not necessary, more for convenience to
administer database

Question 24

Network connections between the SGH Citrix
servers & SCM database were allowed
2

Answer 24

• A basic security review of the network architecture and connectivity
between the SGH Citrix servers and the SCM database could have
shown that the open network connection created a security
vulnerability.
• However, no such review was carried out.

Question 25

Lack of monitoring at the SCM database for

unusual queries and access

Answer 25

From 26 June to 4 July 2018, attacker ran queries on the SCM database,
including bulk queries. Attacker was able to do so unchallenged
because of a lack of monitoring at the SCM database
• there were no existing controls to detect bulk queries being made to
the SCM database.
• there were no controls in place at the time of the attack to detect or
block any queries to the SCM database made using illegitimate
applications.
• database activity monitoring (“DAM”) solutions available on the
market which could address these gaps highlighted above. DAM was
not implemented by IHiS at the time of the attack

Question 26

Lack of monitoring at the SCM database for

unusual queries and access 2

Answer 26

• database activity monitoring (“DAM”) solutions available on the
market which could address some or all of the three gaps highlighted
above. DAM was not implemented by IHiS at the time of the attack

Question 27

SGH Citrix servers were not adequately

secured against unauthorised access

Answer 27

The compromise of the SGH Citrix servers was critical in giving the
attacker access to the SCM database.
• Privileged Access Management was not the exclusive means for
accessing the SGH Citrix servers, and logins to the servers by other
means without 2-factor authentication were possible
• IHiS Citrix administrators not only were aware of this alternative
route, but made use of it form convenience!

Question 28

SGH Citrix servers were not adequately

secured against unauthorised access 2

Answer 28

Lack of firewalls to prevent unauthorised remote access using RDP to
the SGH Citrix servers
• the attacker had moved laterally using RDP to remotely access
multiple SGH Citrix servers.
• This was done from compromised workstations and suspected VM,
and by using compromised user credentials.
• After compromising the SGH Citrix servers, the attacker was able to
connect to Citrix Server 3 in the H-Cloud.
• The attacker also queried the SCM database from Citrix Server 2, a
SGH server.

Question 29

SGH Citrix servers were not adequately

secured against unauthorised access 3

Answer 29

• If RDP access from end-user workstations to the SGH Citrix servers
had been disabled or restricted, it would have made it harder for the
attacker to move laterally and to compromise the SGH Citrix servers.
• However, at the time of the attack, there were no firewalls in place to
prevent unauthorised remote access to the SGH Citrix servers using
RDP

Question 30

Weak controls over and inadequate

monitoring of local administrator accounts

Answer 30

the password to the (dormant) L.A. account was ‘P@ssw0rd’, which is easily cracked, and it is possible that the attacker gained control over the account
by cracking the password.
• The weak password and the fact that the attacker was able to use the
dormant account to access Citrix Server 1 were in spite of three relevant
IHiS policies:
1. user passwords are to be changed periodically. However, the password to
the L.A. account was unchanged from 2012 till 11 June 2018.
2. 2017, IHiS instituted a policy under which administrators were required to have more complex passwords.
3. Dormant or unused accounts should be identified and disabled, in order
to prevent usage in unauthorised activities

Question 31

Lack of sight over and mismanagement of the

S.A. service account

Answer 31

the S.A. account was used by the attacker to access Citrix Server 2, including
when querying the SCM database. The existence of and privileges attached
to the account facilitated this use.
1. there was no real need for the S.A. account to exist, as there was no
actual use in IHiS of the relevant service for which it was created. Yet it
existed on all Citrix servers in which the service had been installed, and
the account had full administrative privileges to login to the server,
including logging in interactively.
2. The Citrix Team did not know of this account!
3. S.A. account was an unused account that should have been disabled

Question 32

Observations on the overall management of

SGH Citrix servers

Answer 32

They were treated as not mission critical, unlike SCM database
• The SGH Citrix servers were not monitored for real-time analysis and
alerts of vulnerabilities and issues arising from these servers.
• Vulnerability scanning, which was carried out for mission-critical
systems, was not carried out for the SGH Citrix servers.
• Vulnerability scanning is an inspection of the potential points of exploit on a
computer to identify gaps in security

Question 33

Internet connectivity in the SingHealth IT

network increased the attack surface 1

Answer 33

• The SingHealth network’s connection to the Internet, while serving
their operational needs, created an avenue of entry and exit for the
attacker. This allowed the attacker to make use of an internetconnected workstation (Workstation A) to gain entry to the network,
before making his way to the SCM database to steal the medical data.

Question 34

Internet connectivity in the SingHealth IT

network increased the attack surface 2

Answer 34

• The security risks arising from internet-connectivity in the SingHealth
network were raised by CSA to MOH from as early as August 2015;
• By June 2017, the healthcare sector had determined, that
• internet access would be removed for staff that did not require the internet
for work,
• for staff that required the internet for work, access would be through a secure
internet access platform which, at that time, was to take the form of a
‘remote browser’.
• When the Cyber Attack occurred, the remote browser solution was
not yet rolled out. IHiS was on the cusp of awarding the tender for the
remote browser solution in July 2018 when the Cyber Attack occurred

Question 35

Internet connectivity in the SingHealth IT

network increased the attack surface 3

Answer 35

1. SGH Citrix servers: At the time of the attack, a user who accessed
   pre-configured internet websites through the SGH Citrix servers
   would be able to access websites other than the pre-configured
   sites simply by keying in the internet URL in the address bar of the
   web browser. If such other websites were malicious, it would be
   possible that malware would be downloaded onto the SGH Citrix
   server.
2. The S.P. server: The S.P. server was detected trying to connect to a
   C2 server on 19 July 2018.

Question 36

Versions of Outlook used by IHiS were not patched

against a publicly available hacking tool

Answer 36

• The attacker was able to install the hacking tool (publicly available) on
Workstation A on 1 December 2017 by exploiting a vulnerability in
the version of the Outlook application installed on the workstation!
• A patch that was effective in preventing the vulnerability from being
exploited (and thus to prevent the installation of the tool) was
available since late-2017!
• Clear need to improve software upgrade policies!

Question 37

Coding vulnerability in the SCM application

Answer 37

CSA’s analysis of the SCM application showed that there were signs of
insecure coding practices, giving rise to a vulnerability that was likely
exploited by the attacker to obtain the credentials to the A.A. account.
• Sep 2014, Zhao, then-employee of IHiS, discovered a method of
exploiting the vulnerability. He reported to his supervisor.
• Vulnerability likely played a pivotal role in allowing the attacker to
obtain the SCM database credentials and cross the last mile to gain
access into the SCM database.
• IHiS has accepted that if further queries and investigations had in fact
been carried out, the coding vulnerability could have been discovered

Question 38

Coding vulnerability –more details

Answer 38

• Supervisor gave evidence that she asked Zhao to log a case with
Allscripts (soln provider), but she did not follow-up with him on
whether he had in fact done so.
• Zhao did not; instead Sept 17, he emailed Allscripts competitor EPIC
abt this vulnerability.
• Allscripts boss David Chambers came to know abt Zhao’s email & he
wrote to CEO IHIS abt it!
• CEO IHIS fired Zhao immediately after verification.
• But no action was taken to investigate alleged vulnerability!

Question 39

Other vulnerabilities in the network that were

identified in the FY16 H-Cloud Pen-Test

Answer 39

• Administrator credentials were found on network shares
• A Citrix administrator password was also found in a Windows batch
file.
• During a scanning process done after the Cyber Attack, a script file
containing credentials for an administrator account was found, which
had the password ‘P@ssw0rd’.
• This was in fact the very same account flagged by the penetration
testers during the FY16 H-Cloud Pen-Test!

Question 40

Other vulnerabilities in the network that were

identified in the FY16 H-Cloud Pen-Test

Answer 40

• The Citrix virtualisation environment was not configured adequately
to prevent attackers from breaking out into the underlying operating
system
• Exploiting the vulnerability allowed the penetration testers to access
files and execute arbitrary commands.
• CSA’s hypothesis is that this vulnerability could have been the means
by which the attacker gained initial access to the file system of any of
the compromised SGH Citrix servers.

Question 41

THE ATTACKER – TOOLS AND COMMAND AND

CONTROL INFRASTRUCTURE

Answer 41

• Customised and stealthy malware –new even to cybersecurity experts
• A variety of custom web shells, tools, and unique malware were used
in the attack. Early-stage tools were used to gain a foothold within the
network. Intermediate-stage tools, including some custom tools, were
used to perform various tasks such as reconnaissance, privilege
escalation and lateral movement.
• Remote Access Trojans, such as the abovementioned RAT 1 and RAT
2, were used to provide the attacker with full control over specific
infected systems and to serve as backdoors to re-enter the network.

Question 42

Extensive C2 Infrastructure

Answer 42

CSA’s forensic analysis revealed a number of network Indicators of
Compromise (“IOCs”) which appeared to be overseas C2 servers. CSA has
explained that generally, the C2 servers were used for:
• Infection: where the server is used as a means of dropping malware into
the system it is trying to infect;
• Data exfiltration: there were indications of technical data being sent to the
servers; and
• Beacon: infected machines may have connected to C2 servers to establish a
‘heartbeat’, which refers to a slow, rhythmic communication meant just to
sustain communications.

Question 43

Actions of COI Committee

Answer 43

The Committee made 16
recommendations, 7 of which are
priority ones, to be implemented
immediately! They are