FOSS 8.8.15 CentOS 7, 24GB RAM, 6 older XEON cores (1st gen Core i7 equivalent), SSD backed storage (OS and hardware upgrades are pending)
1 MTA/LDAP/MAIL STORE server, 1 PROXY server, 1 Z-PUSH activesync server
We have 1 user whose mailbox becomes unresponsive several times a week, other users aren't affected. The temporary fix has been to restart the Zimbra server.
The user has a 65GB mailbox, the mail server serves about 100 users.
All Zimlets on the affected user mailbox have been disabled without improvement. I haven't verified by the issue may be correlated with the user searching their mailbox..
When the issue is occurring I've checked the server and see that MYSQLD is taking up most of the CPU timeAfter the system is restarted via zmcontrol restart then MYSQLD CPU time is almost non-existent.
I've run an older release of mysqltuner.pl while the issue was occurring with the following results (uptime was 4 days)I also ran the current version of mysqltuner.pl but the server had already been restarted:The my.cnf fileAny the Java Heap Size:Any ideas on why the 1 user mailbox has this reoccurring issue?
Thank you!
1 MTA/LDAP/MAIL STORE server, 1 PROXY server, 1 Z-PUSH activesync server
We have 1 user whose mailbox becomes unresponsive several times a week, other users aren't affected. The temporary fix has been to restart the Zimbra server.
The user has a 65GB mailbox, the mail server serves about 100 users.
All Zimlets on the affected user mailbox have been disabled without improvement. I haven't verified by the issue may be correlated with the user searching their mailbox..
When the issue is occurring I've checked the server and see that MYSQLD is taking up most of the CPU time
Code:
top - 11:10:49 up 4 days, 11:07, 1 user, load average: 8.82, 7.46, 6.07Tasks: 242 total, 2 running, 240 sleeping, 0 stopped, 0 zombie%Cpu(s): 66.7 us, 2.8 sy, 0.0 ni, 29.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.9 stKiB Mem : 24522192 total, 677264 free, 13092960 used, 10751968 buff/cacheKiB Swap: 16777212 total, 16775668 free, 1544 used. 9886104 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3740 zimbra 20 0 9824.7m 2.9g 7216 S 387.5 12.2 738:44.62 mysqld 3873 zimbra 20 0 9868.8m 5.8g 18328 S 31.2 24.8 649:47.83 java 17593 zimbra 20 0 58576 2272 1472 R 12.5 0.0 0:00.03 top 9 root 20 0 0 0 0 R 6.2 0.0 12:33.31 rcu_sched 1 root 20 0 46088 5572 3060 S 0.0 0.0 5:16.62 systemd
I've run an older release of mysqltuner.pl while the issue was occurring with the following results (uptime was 4 days)
Code:
-------- Storage Engine Statistics -----------------------------------------------------------------[--] Status: +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA +SEQUENCE [--] Data in MyISAM tables: 4.0K (Tables: 1)[--] Data in InnoDB tables: 1.7G (Tables: 1631)[OK] Total fragmented tables: 0 -------- Analysis Performance Metrics --------------------------------------------------------------[--] innodb_stats_on_metadata: OFF[OK] No stat updates during querying INFORMATION_SCHEMA. -------- Security Recommendations ------------------------------------------------------------------[OK] There are no anonymous accounts for any database users[OK] All database users have passwords assigned[!!] User ''zimbra'@%' does not specify hostname restrictions.[!!] There is no basic password file list! -------- CVE Security Recommendations --------------------------------------------------------------[--] Skipped due to --cvefile option undefined -------- Performance Metrics -----------------------------------------------------------------------[--] Up for: 4d 10h 59m 2s (5M q [13.768 qps], 34K conn, TX: 396G, RX: 593M)[--] Reads / Writes: 89% / 11%[--] Binary logging is disabled[--] Physical Memory : 23.4G[--] Max MySQL memory : 10.4G[--] Other process memory: 0B[--] Total buffers: 8.3G global + 18.8M per thread (110 max threads)[--] P_S Max memory usage: 0B[--] Galera GCache Max memory usage: 0B[OK] Maximum reached memory usage: 8.5G (36.53% of installed RAM)[OK] Maximum possible memory usage: 10.4G (44.30% of installed RAM)[OK] Overall possible memory usage with other process is compatible with memory available[OK] Slow queries: 0% (1K/5M)[OK] Highest usage of available connections: 10% (11/110)[OK] Aborted connections: 0.00% (0/34488)[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance[OK] Query cache is disabled by default due to mutex contention on multiprocessor machines.[OK] Sorts requiring temporary tables: 0% (162 temp sorts / 31K sorts)[OK] No joins without indexes[OK] Temporary tables created on disk: 2% (398 on disk / 14K total)[OK] Thread cache hit rate: 99% (11 created / 34K connections)[OK] Table cache hit rate: 99% (1K open / 1K opened)[!!] table_definition_cache(400) is lower than number of tables(1792) [OK] Open file limit used: 0% (55/524K)[OK] Table locks acquired immediately: 100% (1M immediate / 1M locks) -------- Performance schema ------------------------------------------------------------------------[--] Performance schema is disabled.[--] Memory used by P_S: 0B[--] Sys schema isn't installed. -------- ThreadPool Metrics ------------------------------------------------------------------------[--] ThreadPool stat is enabled.[--] Thread Pool Size: 6 thread(s).[--] Using default value is good enough for your version (10.1.25-MariaDB) -------- MyISAM Metrics ----------------------------------------------------------------------------[!!] Key buffer used: 18.2% (24M used / 134M cache)[OK] Key buffer size / total MyISAM indexes: 128.0M/127.0K[OK] Read Key buffer hit rate: 100.0% (118 cached / 0 reads)[!!] Write Key buffer hit rate: 0.0% (172 cached / 0 writes) -------- InnoDB Metrics ----------------------------------------------------------------------------[--] InnoDB is enabled.[--] InnoDB Thread Concurrency: 0[OK] InnoDB File per table is activated[OK] InnoDB buffer pool / data size: 8.2G/1.7G[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (11.9317503877819 %): 500.0M * 2/8.2G should be equal to 25%[OK] InnoDB buffer pool instances: 8[--] InnoDB Buffer Pool Chunk Size not used or defined in your version[OK] InnoDB Read buffer efficiency: 100.00% (14801483326 hits/ 14801575032 total)[!!] InnoDB Write Log efficiency: 86.02% (174511 hits/ 202869 total)[OK] InnoDB log waits: 0.00% (0 waits / 28358 writes) -------- Aria Metrics ------------------------------------------------------------------------------[--] Aria Storage Engine not available. -------- TokuDB Metrics ----------------------------------------------------------------------------[--] TokuDB is disabled. -------- XtraDB Metrics ----------------------------------------------------------------------------[--] XtraDB is disabled. -------- Galera Metrics ----------------------------------------------------------------------------[--] Galera is disabled. -------- Replication Metrics -----------------------------------------------------------------------[--] Galera Synchronous replication: NO[--] No replication slave(s) for this server.[--] Binlog format: STATEMENT[--] XA support enabled: ON[--] Semi synchronous replication Master: Not Activated[--] Semi synchronous replication Slave: Not Activated[--] This is a standalone server -------- Recommendations ---------------------------------------------------------------------------General recommendations: Restrict Host for 'zimbra'@% to 'zimbra'@LimitedIPRangeOrLocalhost RENAME USER 'zimbra'@'%' TO 'zimbra'@LimitedIPRangeOrLocalhost; Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1 Performance schema should be activated for better diagnostics Consider installing Sys schema from https://github.com/mysql/mysql-sys for MySQL Consider installing Sys schema from https://github.com/FromDual/mariadb-sys for MariaDB Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: https://bit.ly/2TcGgtUVariables to adjust: table_definition_cache(400) > 1792 or -1 (autosizing if supported) performance_schema = ON enable PFS innodb_log_file_size should be (=1G) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
Code:
-------- Performance Metrics -----------------------------------------------------------------------ℹ Up for: 1h 4m 7s (65K q [16.985 qps], 370 conn, TX: 4G, RX: 7M)ℹ Reads / Writes: 82% / 18%ℹ Binary logging is disabledℹ Physical Memory : 23.4Gℹ Max MySQL memory : 8.6Gℹ Other process memory: 0Bℹ Total buffers: 8.3G global + 2.8M per thread (110 max threads)ℹ Performance_schema Max memory usage: 0Bℹ Galera GCache Max memory usage: 0B✔ Maximum reached memory usage: 8.4G (35.90% of installed RAM)✔ Maximum possible memory usage: 8.6G (36.96% of installed RAM)✔ Overall possible memory usage with other process is compatible with memory available✔ Slow queries: 0% (35/65K)✔ Highest usage of available connections: 18% (20/110)✔ Aborted connections: 0.00% (0/370)✘ Name resolution is active: a reverse name resolution is made for each new connection which can reduce performance✔ Query cache is disabled by default due to mutex contention on multiprocessor machines.✔ Sorts requiring temporary tables: 0% (0 temp sorts / 477 sorts)✔ No joins without indexes✔ Temporary tables created on disk: 10% (17 on disk / 162 total)✔ Thread cache hit rate: 94% (20 created / 370 connections)✔ Table cache hit rate: 98% (296 hits / 302 requests)✘ table_definition_cache (400) is less than number of tables (1792) ✔ Open file limit used: 0% (23/524K)✔ Table locks acquired immediately: 100% (15K immediate / 15K locks) -------- Performance schema ------------------------------------------------------------------------✘ Performance_schema should be activated.ℹ Sys schema is not installed. -------- ThreadPool Metrics ------------------------------------------------------------------------ℹ ThreadPool stat is disabled. -------- MyISAM Metrics ----------------------------------------------------------------------------✘ Consider migrating 1 following tables to InnoDB:ℹ * InnoDB migration request for `zimbra`.`service_status` Table: ALTER TABLE `zimbra`.`service_status` ENGINE=InnoDB;ℹ General MyIsam metrics:ℹ +-- Total MyISAM Tables : 1ℹ +-- Total MyISAM indexes : 127.0Kℹ +-- KB Size :128.0Mℹ +-- KB Used Size :23.3Mℹ +-- KB used :18.2%ℹ +-- Read KB hit rate: 100.0% (1 cached / 0 reads)ℹ +-- Write KB hit rate: 0.0% (2 cached / 0 writes)✘ Key buffer used: 18.2% (23.3M used / 128.0M cache)✔ Key buffer size / total MyISAM indexes: 128.0M/127.0K✔ Read Key buffer hit rate: 100.0% (1 cached / 0 reads)✘ Write Key buffer hit rate: 0.0% (2 cached / 0 writes) -------- InnoDB Metrics ----------------------------------------------------------------------------ℹ InnoDB is enabled.ℹ InnoDB Thread Concurrency: 0✔ InnoDB File per table is activated✔ InnoDB Buffer Pool size ( 8.2G ) under limit for 64 bits architecture: (17179869184.0G )✔ InnoDB buffer pool / data size: 8.2G / 1.7G✘ Ratio InnoDB log file size / InnoDB Buffer pool size (11.9317503877819%): 500.0M * 2 / 8.2G should be equal to 25%✔ InnoDB buffer pool instances: 8ℹ InnoDB Buffer Pool Chunk Size not used or defined in your version✔ InnoDB Read buffer efficiency: 99.98% (135954554 hits / 135987768 total)✘ InnoDB Write Log efficiency: 88.9% (4391 hits / 4939 total)✔ InnoDB log waits: 0.00% (0 waits / 548 writes) -------- Aria Metrics ------------------------------------------------------------------------------ℹ Aria Storage Engine not available. -------- TokuDB Metrics ----------------------------------------------------------------------------ℹ TokuDB is disabled. -------- XtraDB Metrics ----------------------------------------------------------------------------ℹ XtraDB is disabled. -------- Galera Metrics ----------------------------------------------------------------------------ℹ Galera is disabled. -------- Replication Metrics -----------------------------------------------------------------------ℹ Galera Synchronous replication: NOℹ No replication slave(s) for this server.ℹ Binlog format: STATEMENTℹ XA support enabled: ONℹ Semi synchronous replication Master: Not Activatedℹ Semi synchronous replication Slave: Not Activatedℹ This is a standalone server -------- Recommendations ---------------------------------------------------------------------------General recommendations: You are using an unsupported version for production environments Upgrade as soon as possible to a supported version ! MySQL was started within the last 24 hours: recommendations may be inaccurate Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=ON Performance schema should be activated for better diagnostics Consider installing Sys schema from https://github.com/FromDual/mariadb-sys for MariaDB MyISAM engine is deprecated, consider migrating to InnoDB Be careful, increasing innodb_log_file_size / innodb_log_files_in_group means higher crash recovery mean timeVariables to adjust: skip-name-resolve=ON table_definition_cache (400) > 1792 or -1 (autosizing if supported) performance_schema=ON key_buffer_size (~ 24M) innodb_log_file_size should be (=1G) if possible, so InnoDB total log file size equals 25% of buffer pool size. innodb_log_buffer_size (> 8M)
Code:
[mysqld]bind-address = 127.0.0.1#innodb_force_recovery = 1basedir = /opt/zimbra/commondatadir = /opt/zimbra/db/datasocket = /opt/zimbra/data/tmp/mysql/mysql.sockpid-file = /opt/zimbra/log/mysql.pidport = 7306user = zimbratmpdir = /opt/zimbra/data/tmpexternal-lockingslow_query_log = 1slow_query_log_file = /opt/zimbra/log/myslow.loggeneral_log_file = /opt/zimbra/log/mysql-mailboxd.loglong_query_time = 1log_queries_not_using_indexesthread_cache_size = 110max_connections = 110# We do a lot of writes, query cache turns out to be not useful.query_cache_type = 0sort_buffer_size = 1048576read_buffer_size = 1048576# (Num mailbox groups * Num tables in each group) + paddingtable_open_cache = 3000innodb_data_file_path = ibdata1:10M:autoextendinnodb_buffer_pool_size = 8788753613innodb_log_file_size = 524288000innodb_log_buffer_size = 8388608innodb_file_per_table# Value is: 200 + max_connections + 2 * table_open_cacheinnodb_open_files = 2710innodb_max_dirty_pages_pct = 10innodb_flush_method = O_DIRECTinnodb_flush_log_at_trx_commit = 0max_allowed_packet = 16777216[mysqld_safe]log-error = /opt/zimbra/log/mysqld.logpid-file = /opt/zimbra/log/mysql.pid
Code:
mailboxd_java_heap_size = 4765
Thank you!
Statistics: Posted by PaperAdvocate — Thu Mar 13, 2025 7:46 pm