Processes Running Deleted Code

What this check and mySites.guru tool looks at on your site

Every other check we run reads your files or your database. This one reads memory, because there is a way of getting through a clean-up that leaves nothing in either. It is one check among many, and like all of them it answers one question rather than telling you whether a site is hacked. Use it alongside the file scans, the crontab and scheduler checks and the administrator audit, not instead of them.

It works like this. Something gets onto the site and starts a PHP program running in the background. That program then deletes its own file and the folder it was started from. Nothing is left behind, so a file scan finds nothing and a scheduled-task check finds nothing. But the program is still running, still holding a copy of the code, and it writes index.php back within seconds of you deleting it.

On every snapshot we look at the programs running under your hosting account and report any that are running code which is no longer on disk. There are three separate things we check for:

• The script has been deleted. The program was started from a file, and that file is gone.
• The folder has been deleted. The directory the program was started in no longer exists.
• The PHP program itself has been deleted. We report this one and never offer to stop it, because your host does exactly this to everything running each time it updates PHP. On its own it usually means your server was patched, not that anything is wrong.

We only ever look at programs running under your own hosting account. On a shared server, other customers’ programs are none of our business, and we stop reading a program’s details the moment we can see it is not yours.

That is a consequence of how we connect to your site rather than a rule we chose to apply on top. mySites.guru runs as an ordinary PHP program under your own hosting account, not as root and not as a privileged agent installed on the server, so what we can see is close to what an attacker who got into your site can see and no more. It is why we can see the malicious program at all, since it was started through your web server and runs as the same user we do. It is equally why a host that hides the list from PHP hides it from us: that setting does not apply to root, and we are not root, on purpose.

Why a program you cannot see beats a clean-up

The reason this matters is timing. Most reinfections take an hour, because they are driven by a scheduled task waiting for its next run. This one takes seconds, because the thing rewriting your files never stopped.

That changes what the symptom looks like. A site owner deletes a webshell, refreshes the page, and it is already back. It reads as though the deletion did not work, or as though the file was never really removed, so the natural next step is to delete it again, harder, and then to start doubting the tooling. We have watched people go round that loop for an afternoon. Nothing they did was wrong. The file really was deleted every time, and something still running put it back before they finished looking.

It also defeats the usual advice. Restoring a backup replaces your files and your database, and does nothing at all to a program that is already running in memory, so the reinfection outlives the restore and starts again. Changing every password does not help either, because the program does not need to log in. It is already inside, and it will keep writing until the process ends or the server restarts.

The related case worth knowing about is the scheduled task, covered in reinfected? check every crontab, not just yours. The two often appear together, and they need removing in the right order: stop the running program first, then remove the schedule, then clean the files. Clean the files first and you have simply given both of them something to undo.

Every program accounted for

A clean result means every PHP program running under your hosting account is still running code that exists on disk. Your CMS’s own background work, a queue worker, a backup tool, a scheduled import: whatever is running, the file behind it is still there and can be pointed at.

That is narrower than it sounds, and we would rather say so than overstate it. It covers the programs running under your account at the moment we looked, on a server that let us look. It is not a statement about the whole machine, and it is not a statement about what was running an hour ago.

How to fix it

1. Read the row before you act on it. Each one shows the command line, how long the program has been running, and which of the three things is missing. Something you set up will usually be recognisable from its command line, and something you did not set up will usually not be.
2. Stop the program before you touch the files. This is the part people get backwards. Delete the files while it is still running and it puts them back, often before you have finished.
3. Then remove whatever restarts it. A running program is one half of the persistence. Check your scheduled tasks as well, both the server’s own and your CMS’s, or it comes back on the next run.
4. Then clean the files, and check what let it in. A program running from a deleted file is a symptom. Something put it there, and until you find that, the same route is still open.
5. If you cannot stop it from here, use SSH or ask your host. On a server that does not let us look, or where the evidence is too weak to act on safely, the process list over SSH will show you the same thing.
6. Check your other sites on the same account. Anything that could start a background program on one site can usually reach the others beside it.

What mySites.guru does about it

On every snapshot we look, and we tell you which of the three answers you got: we looked and found nothing, we looked and found something, or we could not look and here is why. That third answer is the one most tools skip, and it is why a grey badge here never means clean.

Where we can act, you get a Stop button on the row itself, and every one of the rules behind it is checked again on your server at the moment you press it rather than trusted from the page you are looking at. It must still be running under your account, still unattended, still running code that is gone, and still traceable to this site. It also has to still be the same program: process numbers get reused, so if anything has changed since we showed you the row, we will not act, and ask you to reload rather than stopping whatever now holds that number.

After a stop we look again and tell you what changed, because ending a program does not end anything it started. If it left something behind, the second look finds it.