Quantcast
Viewing all articles
Browse latest Browse all 1145

Administrators • Random HTTP ERROR 504 Upstream Server Issues

Single server install
Version : 8.8.15P33
CentOS 7

NOTE : Yes, this is out of date and it is on the project list to migrate to an updated server.

Couple days ago my users started getting the following error at random.
Image may be NSFW.
Clik here to view.
Screenshot from 2025-01-30 10-24-18.png
I am seeing the following in the logs

Code:

2025/01/29 20:29:41 [error] 11491#0: *2527 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xxx.xxx.xxx, server: zim.xxx.com, request: "POST /service/soap/GetMsgRequest HTTP/2.0", upstream: "https://127.0.0.1:8443/service/soap/GetMsgRequest", host: "zim.xxx.com", referrer: "https://zim.xxx.com/"
The database is only 8GB in size and the users number in the low hundreds, but most of them aren't using the service. I have 32GB of RAM on the server and it's currently using 12GB. There are eight cores but at most we only hammer three at a time, and that appears to be mysql threads but it's not consistent.

The MySQL slow transaction log shows the following

Code:

# Time: 250130  7:59:06# User@Host: zimbra[zimbra] @ localhost [127.0.0.1]# Thread_id: 4016  Schema: zimbra  QC_hit: No# Query_time: 0.000203  Lock_time: 0.000055  Rows_sent: 0  Rows_examined: 0# Rows_affected: 0SET timestamp=1738241946;SELECT mailbox_id, item_id FROM pending_acl_push WHERE date < 1738241946983;# Time: 250130  8:14:06# User@Host: zimbra[zimbra] @ localhost [127.0.0.1]# Thread_id: 4019  Schema: zimbra  QC_hit: No# Query_time: 0.000201  Lock_time: 0.000056  Rows_sent: 0  Rows_examined: 0# Rows_affected: 0SET timestamp=1738242846;SELECT mailbox_id, item_id FROM pending_acl_push WHERE date < 1738242846983;# Time: 250130  8:19:53# User@Host: zimbra[zimbra] @ localhost [127.0.0.1]# Thread_id: 4017  Schema: zimbra  QC_hit: No# Query_time: 2.829178  Lock_time: 0.018193  Rows_sent: 1  Rows_examined: 12351# Rows_affected: 0SET timestamp=1738243193;
Additonally there is another long running query that looks like this

Code:

...824 AND mi.id <= 892857)) OR ((mi.id >= 892886 AND mi.id <= 892891)) OR ((mi.id >= 892895 AND mi.id <= 892915)) OR ((mi.id >= 892923 AND mi.id <= 892925)) OR ((mi.id >= 892927 AND mi.id <= 892930)) OR ((mi.id >= 892949 AND mi.id <= 892951)) OR ((mi.id >= 892954 AND mi.id <= 892960)) OR ((mi.id >= 892962 AND mi.id <= 892965)) OR ((mi.id >= 892972 AND mi.id <= 892973)) OR ((mi.id >= 892975 AND mi.id <= 892976)) OR ((mi.id >= 892993 AND mi.id <= 892999)) OR ((mi.id >= 893001 AND mi.id <= 893002)) OR ((mi.id >= 893007 AND mi.id <= 893010)) OR ((mi.id >= 893020 AND mi.id <= 893022)) OR ((mi.id >= 893025 AND mi.id <= 893029)) OR ((mi.id >= 893040 AND mi.id <= 893075)) OR ((mi.id >= 893123 AND mi.id <= 893131)) OR ((mi.id >= 893136 AND mi.id <= 893143)) OR ((mi.id >= 893145 AND mi.id <= 893146)) OR ((mi.id >= 893148 AND mi.id <= 893158)) OR ((mi.id >= 893170 AND mi.id <= 893177)) OR ((mi.id >= 893179 AND mi.id <= 893181)) OR ((mi.id >= 893184 AND mi.id <= 893187)) OR ((mi.id >= 893189 AND mi.id <= 893191)) OR ((mi.id >= 893196 AND mi.id <= 893202)) OR ((mi.id >= 893204 AND mi.id <= 893214)) OR ((mi.id >= 893216 AND mi.id <= 893218)) OR ((mi.id >= 893220 AND mi.id <= 893228)) OR ((mi.id >= 893231 AND mi.id <= 893232)) OR ((mi.id >= 893237 AND mi.id <= 893241)) OR ((mi.id >= 893243 AND mi.id <= 893254)) OR ((mi.id >= 893256 AND mi.id <= 893262)) OR ((mi.id >= 893265 AND mi.id <= 893267)) OR ((mi.id >= 893269 AND mi.id <= 893270)) OR ((mi.id >= 893274 AND mi.id <= 893281)) OR ((mi.id >= 893283 AND mi.id <= 893322)) OR ((mi.id >= 893324 AND mi.id <= 893334)) OR ((mi.id >= 893337 AND mi.id <= 893338)) OR ((mi.id >= 893340 AND mi.id <= 893342)) OR ((mi.id >= 893345 AND mi.id <= 893348)) OR ((mi.id >= 893352 AND mi.id <= 893353)) OR ((mi.id >= 893356 AND mi.id <= 893366)) OR ((mi.id >= 893368 AND mi.id <= 893380)) OR ((mi.id >= 893382 AND mi.id <= 893391)) OR ((mi.id >= 893416 AND mi.id <= 893417)) OR ((mi.id >= 893419 AND mi.id <= 893421)) OR ((mi.id >= 893423 AND mi.id <= 893429)) OR ((mi.id >= 893431 AND mi.id <= 893432)) OR ((mi.id >= 893435 AND mi.id <= 893439)) OR ((mi.id >= 893445 AND mi.id <= 893448)) OR ((mi.id >= 893453 AND mi.id <= 893459)) OR ((mi.id >= 893461 AND mi.id <= 893478)) OR ((mi.id >= 893483 AND mi.id <= 893489)) OR ((mi.id >= 893507 AND mi.id <= 893508)) OR ((mi.id >= 893515 AND mi.id <= 893516)) OR ((mi.id >= 893518 AND mi.id <= 893520)) OR ((mi.id >= 893525 AND mi.id <= 893528)) OR ((mi.id >= 893532 AND mi.id <= 893542)) OR ((mi.id >= 893544 AND mi.id <= 893545)) OR ((mi.id >= 893550 AND mi.id <= 893552)) OR ((mi.id >= 893559 AND mi.id <= 893561)) OR ((mi.id >= 893568 AND mi.id <= 893571)) OR ((mi.id >= 893573 AND mi.id <= 893575)) OR ((mi.id >= 893583 AND mi.id <= 893587)) OR ((mi.id >= 893615 AND mi.id <= 893619)) OR ((mi.id >= 893621 AND mi.id <= 893626)) OR ((mi.id >= 893629 AND mi.id <= 893635)) OR ((mi.id >= 893638 AND mi.id <= 893654)) OR ((mi.id >= 893657 AND mi.id <= 893664)) OR ((mi.id >= 893667 AND mi.id <= 893668)) OR ((mi.id >= 893675 AND mi.id <= 893681)) OR...
I have played with the proxy settings, timeouts, database cache, and good old fashion reboots. Nothing has resolved this problem.

Any ideas?

Statistics: Posted by domadmin — Thu Jan 30, 2025 3:42 pm



Viewing all articles
Browse latest Browse all 1145

Trending Articles