Explanation
|
Video and Pics
|
Provider / Hardware Issues
Hetzner Internet Down
SNBCHF and all sites down. Dec 20, 2017.
Hetzner down. The downtime was 20 minutes
|
We can’t do anything in these cases. They do not provide any details for the problem. |
All sites down
SNBCHF and all sites down. Jul 02, 2019.
After a 2 hour investigation early in the morning, the temperature of the server was to high and it stop responding. This was a hardware problem. In these cases, we need to contact support and explain what is the problem. You send them a ticket using a special support page.
Hetzner usualy needs 5 – 15 minutes to take care about the server, if you give them a proper brief what is the problem. In our case it was the CPU Fan. They give you the reason of the problem and what were the solution by email. |
|
Harddisk Broken
One of the hard drives of the server was broken. Hetzner replace it with new one. Then we did a sync. |
|
Low Space
No Space left on disk
When we store many backups on the server, the disk space is full and the site stops working. The site needs always some disk space for properly working.
The problem: We simply need to delete the old backups from the server.
Deletion can be done via BackupBuddy page or via FTP browsing wp-content/uploads. Then we need to delete the old backups and after 1 minute all the site services goes up again.
Important!
If we want this issue to stop appear, we should add this task into Biweekly. Rad should monitor the disk space and delete backups from both of the servers.
|
No Disc Space - Click to enlarge Delete Backup - Click to enlarge |
Purging cache
Yes, It is not good idea to purge our caches everyday.
Do not purge cache every day. |
Purge cache - Click to enlarge |
High CPU / Resources
Usually during the Crons from
Slow Template
Too many plugins: They make high activity. |
Get rid of some plugins or improve template.
|
Many sites
When we have too many sites, then the performance may get bad.
Remove sites.
|
|
Yoast Term Optimizer
Yoast Term Optimizer does not work
It stops working due to lack of enough resources for this single process
The system guy makes the memory, executing time and other resources higher for single process.
|
Yoast Term Optimizer Resourcese Problem - Click to enlarge |
Too Many Crons
We currently have an issue with the server again. Due to too many crons.
- Lack of resources due to too many crons running
- Stop some of them
- Review the crons
Which Crons got deactivated –> Crons page |
Crons SNBCHF - Click to enlarge |
CDN purge
Our server problem on 07/20 was caused by our CDN. We purged the whole cdn cache and their crawlers start caching all our static content again. The problem was that they started too many connections to the our server. The traffic was almost 800% higher than the usual traffic. This is problem of the CDN, they must send regular queries for fetching our content. This traffic is not seen by the Google Analytics, because it is directly to our CSS files. Nothing related with the site. So Google Analytics can track only when the code is installed on that page. It can’t track the traffic to our static files.
We fixed that issue changing the apache and nginx limits to higher. Now we can handle MaxCDN traffic. But it is good idea to purge only some files, not the whole cache. It will save us so many unnecessary queries. |
 Server load at that time on 07/20 - Click to enlarge |
|
We had a performance problem caused by missing pagination |
We introduced pagination, for more see here. |
External Activities
Bad bots on ecoblogs
I was looking on the server logs last two days and see that the performance issues are from bots or bad users from China and Turkey. About 10 Ip addreses are making 70% of all performance. They are making about 10 queries per sec. We need very urgent to revise the logs, get all bad bots and users and ban from our sites. This will lead to better performance, too. |
Bad bots on ecoblogs - Click to enlarge |
Bad Chinese bots
date: June 17, 2020
All sites are not responding. Services like Apache, MySQL, NGNIX do not work any more. Possibly Denial of Service attack.
We notice Chinese bots when we look at the statistics Unresolved IP that open the website. Currently the most visited ip addresses are ours. So no problem.
These IP addresses are banned via htaccess file. |
Bad Chinese bots . |
How to ban IP addresses via htaccess
|
| Block what |
|
|
| specific IP address |
Deny from 123.123.123.123 |
|
| specific domain |
Deny from 123.123.123.123/255.255.255.0 |
|
| multiple IP addresses |
Deny From 111.111.111.111 222.222.222.222 333.333.333.333 |
|
| Block entire subnet |
Deny from 123.123 |
|
|
|
Then I start looking what is causing the problem. The problem was caused by economicblogs.ch and one infinity loop caused by one avatar.
The problem was because the name of the image contains utf-8 special symbols #U2013 and this was the reason for infinity loop. We changed the avatar with different and everything goes to normal. |
|
Server improvements
Server update for better improvement
Hetzner microcode of the processor
As an PX61, PX61-SSD or PX61-NVMe Dedicated Root Server client, we are
writing to inform you that recently a problem was found in the microcode
of the processor used in these models.
The following servers may be affected:
PX61-NVMe #681263 (88.99.241.248) – SNBCHF and Blogs
For further information on this topic, please check the following
resources:
Debian
Tom’s Hardware
Ars Techinca
Based on current information, the problem is triggered by specific
micro-architectural command sequences on Intel’s 6th or 7th generation
CPUs with Hyper-Threading enabled. It can, in rare cases, cause
unpredictable behavior and, in more rare cases, data loss.
|
You can fix the problem by installing the latest Intel microcode update. For
several Linux distributions, you can find the necessary microcode update in the
repositories of your distribution. You can find more information regarding this
topic in our company wiki. (Please see link below.)
If you are using any other operating system (e.g. Windows), or if there
is no update for your Linux distribution yet, you can install the
microcode update by updating your BIOS.
Option: Bios Update by Hetzner
If you would like a BIOS update, please follow these steps:
Write a support ticket for every server that is affected. If you have a preferred time for
when our technicians should perform the update, make sure to include that in
the ticket. You may experience up to 30 minutes of downtime. In the support
ticket, you need to also clearly state that you agree to the downtime.
As an alternative, you can do the BIOS update yourself by following the
instructions in our company wiki:
If you perform the update on your own, and you cannot reach your server for
more than 10 minutes after completing the update, please do NOT perform a
reset. Instead, please write us a support ticket.
Summary:
- We did a BIOS update, because old version of BIOS.
- microcode update on the server
Estimate: 3 hours
Effort: 3 and a half hours |
Weak Resources
How can they have 90 million articles without performance issues?
beforeitsnews.com
onenewspage.com
- Bigger servers. This is very important. They have very big servers, so the performance issues are not a problem for them.
- Ecoblogs is slow: so why don’t we have bigger servers?
We can have big servers, but this will costs more money per month.
- Cache (see W53)
- Template and plugins (see below). What does this mean?
|
Buy bigger server.
|
Missed scheduled posts
Due to some bugs in WP cron, some of the posts were not published. |
Missed scheduled posts . Posts Status . |
|
We installed the WP Missed Schedule plugin taking advice from this article.
They suggest to use this plugin for such a problems and the plugin will take care of everything. We will keep it for a while and if it is useful, it will remain. |
WP Missed Schedule Posts . |
Responsible 
