Lsof
From Dikapedia
$ lsof | grep -i del
.
. (Output cut short)
.
apache2 151110 www-data DEL REG 0,1 81229715 /dev/zero
apache2 151110 www-data DEL REG 0,1 74060991 /dev/zero
apache2 151110 www-data 8w REG 259,2 12017916902 512005 /var/log/apache2/access.log (deleted)
apache2 151110 www-data 9ur REG 259,2 0 5925 /tmp/.ZendSem.SLoNkY (deleted)
apache2 151112 www-data DEL REG 0,1 81229715 /dev/zero
apache2 151112 www-data DEL REG 0,1 74060991 /dev/zero
apache2 151112 www-data 8w REG 259,2 12017916902 512005 /var/log/apache2/access.log (deleted)
apache2 151112 www-data 9u REG 259,2 0 5925 /tmp/.ZendSem.SLoNkY (deleted)
- Its possible a running process opened a file (likely a log file) and kept it locked. At some moment that file was deleted (maybe a wrong logrotate or something) but the process is still running, consequently the file descriptor is kept. You can run: lsof | grep -i del. and check for processes with deleted files (I don’t remember if they will be flagged as deleted, DELETED, delete, etc…). if you find any, confirm if the path matches then we can make a decision (likely kill the process, restart the service that owns it, etc). yep, so the quick fix here would be running service httpd restart (or service apache2 restart depending on the system). something else deleted the files and didn’t let apache know. I would say that it were wrong logrotate rules, but it is just a guess. sometimes a full restart isn’t required, just a apache reload may release the file descriptors and free space. the reload keeps the connections established, so shouldn’t cause downtime (I never promised that though)
- lsof +L1; Check if there are open files which have been deleted. Yeah these files still take up space but don't show in directory. Yep. Need to restart that process to make them go away for good. Needs better log rotating to avoid this issue. yeah when you rm a file, you just mark it for deletion. If its not open, it will delete instantly. But if a process still has the file open, it won’t be truly deleted until that process exits or releases the file.
- Sometimes the OS holds onto a file even if its deleted. lsof +L1 (https://serverfault.com/questions/232525/df-in-linux-not-showing-correct-free-space-after-file-removal/232526).you can try to restart apache. they might need to look into their application on why its locking deleted files.
How to check what's listening on a port
# lsof -i -P -n | grep LISTEN systemd-r 794 systemd-resolve 13u IPv4 16800 0t0 TCP 127.0.0.53:53 (LISTEN) sshd 1024 root 3u IPv4 19859 0t0 TCP *:22 (LISTEN) sshd 1024 root 4u IPv6 19870 0t0 TCP *:22 (LISTEN) java 1223 aws-replication 66u IPv6 28058 0t0 TCP *:1500 (LISTEN)
(Alternate) You can also use netstat:
# sudo netstat -tulpn | grep LISTEN tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 794/systemd-resolve tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1024/sshd tcp6 0 0 :::22 :::* LISTEN 1024/sshd tcp6 0 0 :::1500 :::* LISTEN 1223/java