Host trouble fix

Today there will be a small reminder of what to do on the host if the occupied space is exceeded.

So, we have CentOS 7, where when the space is exceeded, everything starts to work incorrectly.

First, we connect to the SSH in any way and use a simple command to see what exactly happened to us.

find / -type f -size +100M 2>/dev/null

This will show you a list of problematic files. In the simplest case, these could be logs that can be easily and quickly deleted. But they could also be database tables.

In some cases, cache tables in managed databases can grow to enormous sizes, which is a significant problem, because this can lead to such surprises as the inability to clean tables via phpmyadmin, not to mention cleaning via the site admin.

In my last case, the mysql service itself shut down and wouldn't come back to life. So here are my steps that helped revive it and eventually clean up the tables.

The first thing we do is to clear at least 2 GB of space on the server using any methods. You can download any site files and then upload them to the place (while mysql is turned off, the site is not working either, so we will not see any difference).

After we have at least some space, let's look at the status mysql

sudo systemctl status mysqld

to which I personally got several errors

● mysqld.service - MySQL 5.7 database server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Sun 2025-02-23 21:04:04 MSK; 25s ago
  Process: 12194 ExecStopPost=/usr/libexec/mysql-wait-stop (code=exited, status=0/SUCCESS)
  Process: 12162 ExecStartPre=/usr/libexec/mysql-prepare-db-dir %n (code=exited, status=1/FAILURE)
  Process: 12140 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)

Feb 23 21:04:04 515277- systemd[1]: Failed to start MySQL 5.7 database server.
Feb 23 21:04:04 515277- systemd[1]: Unit mysqld.service entered failed state.
Feb 23 21:04:04 515277- systemd[1]: mysqld.service failed.
Feb 23 21:04:04 515277- systemd[1]: mysqld.service holdoff time over, scheduling restart.
Feb 23 21:04:04 515277- systemd[1]: Stopped MySQL 5.7 database server.
Feb 23 21:04:04 515277- systemd[1]: start request repeated too quickly for mysqld.service
Feb 23 21:04:04 515277- systemd[1]: Failed to start MySQL 5.7 database server.
Feb 23 21:04:04 515277- systemd[1]: Unit mysqld.service entered failed state.
Feb 23 21:04:04 515277- systemd[1]: mysqld.service failed.

Next you need to go look at the logs

sudo tail -n 50 /var/log/mysqld.log 

or

sudo journalctl -u mysqld -xe

But in our case, the logs showed nothing at all, so we tried to reset the server.

sudo systemctl reset-failed mysqld

and launch

sudo systemctl start mysqld

But it didn't help, again errors, among which were

touch: cannot touch '/var/log/mysqld.log': Permission denied

Which means that log mysql is not being created. So let's try to create it manually and set the permissions.

sudo touch /var/log/mysqld.log
sudo chown mysql:mysql /var/log/mysqld.log

After that, we reset the service and start it again.

sudo systemctl reset-failed mysqld
sudo systemctl start mysqld

If we don't see any errors in response, it means that our mysql has been successfully launched and we can launch phpmyadmin, clean the tables, and return the files that we decided to temporarily donate to the server.