Ooops. The worst case scenario has come to pass: a vulnerability somewhere allowed a malicious individual (or bot, more likely) to take over your site. Some good news: you've noticed that they did it. Bad news: now you have to clean up.
This guide will give you a series of steps for how to deal with the situation. It may not be exhaustive. If you feel it's missing something, please contribute. The ideas are presented in chronological order. At each step, you will have to make decisions and take actions that are most appropriate for your situation.
Once you are sure that you've been hacked you should stop everything and make a forensic copy of it. Some people prefer to literally yank the network cable and power cable from the server, though that obviously isn't always appropriate and doesn't work for all environments (cloud, anyone?). If you can, this forensic copy could be an operating system level snapshot of the server(s) involved. Otherwise, go for a copy of the database and files. Store one copy to media that can't be modified like a CD or DVD.
"Keeping" a site requires you to spend a lot of time and effort reviewing it. You may decide, depending on the nature of the site, that it is perfectly acceptable to throw it away or rebuild it. Perhaps the needs of the site have shifted and you were planning to rebuild it already. Perhaps it was for an event that has come and gone and you can make a static copy or just delete it all. Even if you've decided to rebuild or delete a site you aren't done with this process, but at least you've cut out a lot of the work in remediation. You may not be able to make this decision or may change your mind based on investigation you do as part of remediation. Start the process by considering this question and it may help ease your life. If you know the specific date that your site was hacked, you may be able to rebuild the site easily simply by using an older database and file backup (you make those and keep them around, right?).
One of the first things you should do is open two scratch pads. One for documenting everything you've discovered and that you're doing NOW. If you notice an important detail proving how the attackers got in, add the source of the information and as much of that information as you can. The second document is for things you want to do to harden your site that you think of as you go through the process. You may realize a lot of things you can do to make it harder for attackers, but their not easy or critical enough to do right now. Save those valuable ideas for later.
You should start deciding who you need to notify about the issue. If your site had users and you believe your site was completely taken over then private content like their email address, IP address and anything else private on the site is compromised. In some locations you are legally bound to inform people of this exposure (e.g. in the case of sites requiring HIPAA or PCI Compliance), if not also morally bound.
If you are not the owner of the site, you should consider which stakeholders to inform. Visitors of the site might have been exposed to malware. The owner should be involved in decision making.
Depending on the nature of the site and who you suspect attacked it, you may wish to notify one or more law enforcement groups. Many local law enforcement groups are poorly equipped for dealing with these kinds of issues. But perhaps they will be able to help or refer your case to another law enforcement group.
Again, depending on the nature of the site you may wish to take it offline. If you suspect that it is actively being used to distribute malware, send spam, or as a pivot point for further attacks then taking it offline and installing a placeholder will at least prevent further damage.
Note that taking the site offline likely does tip the attackers that you are aware of their presence.
If you don't take the site offline at the webserver level:
-
Make sure you've got your forensic copy and then delete out all sessions.
-
If you suspect passwords have been changed, you can update them to new values using a query like this:
update users set pass = concat('ZZZ', sha(concat(pass, md5(rand()))));
Users will be able to use the password reset tool to change their password.
Whatever tipped you off to the fact that you were compromised is the first step in investigating the attack. If your page content was changed, spam emails were sent, information stolen and used for some purpose, malware uploaded to the site, whatever the compromise is - consider how the attacker could have achieved that. List out every way you can think of that someone could make that change. There are two goals to this exercise: first, you'll help pin down how you got attacked so you can close that hole. Second, you'll identify other things the attacker could have done based on the means they used to attack your site. Try to be creative and don't throw out any possibilities as "too hard" or "technically impossible" - your attacker was probably creative and your site may not work exactly as you think it does.
Once you have a list of ways, start trying to pin down which is most likely. Which requires the least effort? What evidence would there be for each one? Do you see examples of that evidence? For each method, is the attack a reasonable thing for someone to do if they had that kind of access? Does one require so much effort that it's not worth it?
When in doubt, ask for help from a colleague that may have more experience with IT and web application security. Sometimes the simple excercise of talking it through with another person will surface additional ideas. Additionally, you should review the OWASP Top 10 list to make sure you're aware of all the various types of attacks. Finally, you find it helpful to search the internet to see if there are others experiencing a similar breach. Often times these attacks are performed by bots that scan a large quantity of websites, so there may be some known signatures to look for as well as solutions to correct and/or mitigate the vulnerability.
If you're lucky, one or two of the potential ways to break in will be the obvious choice. So, block that weakness!
Just like you have a workflow for building a site, attackers have workflows for attacking a site. Loosely speaking, the first step that matters to you is when they have broken into the site. There is often a software bot and/or group of people who are working on this piece and they are not necessarily the people who will exploit the site. Once they have established a foothold, the next step may not come for days or months.
Next is learning about the site and deciding what to do with it. If the site houses a large number of accounts or sensitive data then that is a resource. If it's on a big private network then that becomes something new to explore and exploit. Maybe it's just a clean IP to use to send spam. Maybe it gets a lot of traffic and is a good point to use to distribute malware. If it's got a high search engine rank it can be a place from which to link to other sites and earn search-engine reputation.
Once the potential value of your site is understood it's time to start exploiting it. Attackers may do multiple things at the same time: start by exfiltrating the email/password list and then leave some malware and search-engine links. If your content is particularly sensitive they may copy it and then clean up their tracks so you don't suspect any data was stolen.
If you see signs of a break-in but not of any damage, perhaps you stopped the process early, or perhaps they abused the site in ways that don't leave traces and then haven't done anything else.
A common hack is simply to modify the index.php or any code file in the site such as a template file. The methods are various:
- A virus on the computer used to administer the site which uses stored credentials in a FTP tool to edit and upload the files (seriously).
- Arbitrary code execution on the server and loose server file permissions used to edit or overwrite a file.
- Arbitrary file upload which was used to upload a command shell which was then used to modify the code.
Compare all code files to known good copies, either in the revision control system or from drupal.org (the hacked! module can help with that.)
Look for files on the server that are NOT part of your known Drupal codebase, e.g. modules/system/qseboj.php
Review the files in the "files" directory to ensure they are all appropriate.
It may be helpful to review the combined metadata of owner, group, permissions and timestamps as a fingerprint of the files on the server. If most of the files have one fingerprint and a single other file has a different fingerprint (e.g. edited about when the attack started) that can help you understand what happened.
The php.module in core is really handy for site admins to put a little snippet of code into the Drupal interface to make a tweak to the behavior of a node or block. It's also a great way for an attacker to run whatever code they want to.
Review the contents of all nodes, blocks, user profiles, fields, etc. If you can't legitimately do this by hand then search for malicious content.
The Views UI import or Rules module or CDN or Google Analytics or a few dozen other very popular modules all let you execute PHP via their interfaces. Ouch. These are rarely the ongoing way a site is exploited, but can be useful in pinpointing the point that an attacker came in. Review the ways to execute PHP in your site interface and look in the webserver logs for visits to those URLs. Are the visits from known IP addresses? Are they outside of normal working hours? Knowing the time and IP of your attacker may help you tie together more clues about the attacker from looking in the database at the watchdog.hostname or comment.hostname.
Look at all the roles on your site. What permissions do they have? Which have advanced permissions that could be used to take over a site? Perhaps there is a new role you know you didn't make. Is there one user with that role? It's probably got an email and/or password controlled by the attacker. With your list of roles that have advanced permissions, take a look at the users on your site. Do you know every single one that has advanced permissions? While you're at it: audit the users and remove advanced roles from people who might not need them. Block users who are no longer involved in administering the site.
As you look at all the users on the site, be sure to look not only at their username, but also their email address. Does it make sense for that person? If you're not sure, block the account and double-check with the person. One trick you can use to see if the email has changed since the user registered is to look in the database and compare users.init with users.mail.
Special note: While you should look in the database to help make decisions, if your site has been well and truly compromised remember that the attacker may have modified any row or column in the database. Use known-good backup copies of the database (if there are any) to validate theories.
Many attacks leave traces behind, but a session only lasts until the user clicks logout. Using your list of roles with advanced permissions, look for all sessions associated with users who have those roles. Does the IP in their sessions.hostname make sense for them? Do you see an open session for a user who you think is no longer involved in the organization? Do you see a session for a brand new user that you didn't create? Check the amount of time between when the users logged into the site in the users.login value and their most recent page visit in the session.timestamp: select (s.timestamp - u.login) / 60 / 60 / 24 AS days_since_login, u.uid from sessions s inner join users u on s.uid = u.uid; An attacker may insert a session for a user who hasn't logged in for a long time, making it easy to spot their session.
Drupal's menu_router table gets accessed fairly early in the page request cycle and is a great place to plant some attack code. If you've accidentally or purposefully cleared the menu cache since starting the investigation you may have lost some information from this table as it gets rebuilt from the code on your site in that process. Now aren't you happy you made a forensic copy at the beginning?
One easy solution to this laborious task is to take your forensic copy and then compare it to a version of the menu_router table after you've rebuilt that table. If there are any new rows, missing rows, or rows with different content that deserves investigation. You can also look for common signatures of an attack such as "file_put_contents" or "assert" in the access callback.
If someone has control, including the ability to execute arbitrary php code, they can potentially extend their control to other sites served from the same server or same network. If you can't find any indications of how an attacker breached your site, it's time to consider other avenues like other sites on the server or network. Similarly, if your site is breached it's a good practice to advise others on the server (e.g. via your system administrator or hosting company) so they can review any issues that may have cropped up.
Think about the sensitive information inside your site: Email addresses, password hashes, client IP addresses from watchdog and comment tables, any nodes that were unpublished or hidden from public access in another way. But what else? Drupal has several variables internally that are private: settings.php contains the database authentication information and probably $drupal_hash_salt, both of which are sensitive private information. The drupal_hash_salt is used for a variety of security-related purposes and having that piece of information can make it easier to break back into a site. Sensitive information can also exist inside the variables system (called in code with variable_set and variable_get, usually stored in a database table called variable). The drupal_private_key, in particular, is an important piece of secret information that should be deleted (it will be regenerated the next time it is needed). If your site uses 3rd party APIs such as Facebook connect or Mollom then the API-keys you use to authenticate to those services are compromised. Compromising those 3rd-party keys may make your site less secure and expose your account with those services. If your site allows users to login with OAuth or similar services that provide long-term tokens, then the OAuth tokens stored in your site should be considered compromised.
How do you fix this?
- Replace all sensitive secrets - generate a new $drupal_hash_salt, work with 3rd party tools to invalidate old keys and create new keys, create new credentials for your database, etc..
- Where possible, limit the capability associated with any 3rd-party sites (e.g. if you don't need access to post to Facebook on behalf of that person then don't ask for that permission when you get their FB Connect token).
Depending on what you've found it may be time to revisit your initial decision. If the access the attacker gained was extensive and the changes they made disperse...it can be hard to be confident in bringing the site back online.