Hello, dear forum members! I need your help with a Zimbra issue. After two weeks of unsuccessful attempts to restore the service, you are my last hope. I would greatly appreciate any assistance! Thank you.
Here’s what we have:
•Ubuntu Server 16.04.3 LTS
•ZCS 8.7.11.GA.1854.UBUNTU16.64 (FOSS edition)
•An old (2020) snapshot with a working version of Zimbra
•A new (2025) snapshot with a broken version of Zimbra
The Problem:
At the beginning of this year, something went wrong with Zimbra, and I can't pinpoint exactly what happened. We experienced power issues in our server room, and the TLS certificate for Zimbra expired. I tried to recreate it using Certbot and acme.sh. At the same time, the Zimbra Collaboration Suite also expired. It’s possible that one of these issues, or a combination of them, caused the problem.
It all began when emails stopped being sent (only within the local network). I then discovered the expired certificate and made several unsuccessful attempts to recreate it. I also noticed that the contents of the /opt/zimbra/bin directory were missing. I ran ./install.sh to regenerate the system, but it only made things worse. Now, the system won’t start, and I get the following error:
Additionally, there is another typical error:
However, the service can start separately with the command ldap start.
At this point, I compared the configuration of the old working Zimbra from 2020 [attachment: 2020-old-n-working-conf.txt] with the new broken Zimbra [attachment: 2025-new-n-broken-conf.txt] and found that many variables, especially those related to LDAP and MySQL (where most errors occur), differ, including passwords. I suspect this is part of the problem, but I’m unsure how to fix it. I updated the certificates, fixed permissions using zmfixperms, and even tried importing the output of zmlocalconfig -s from the old version to the new one, but I still encountered startup errors (maybe I missed something).
The broken Zimbra won’t let me change the ldap_root_password with the command (even though it seems this password is stored in plain text):
Attempt to Import Databases:
At one point, I decided to stop trying to fix the broken system and instead transfer the MySQL and LDAP databases from the broken Zimbra to the old working version. Interestingly, I was able to successfully obtain the MySQL database dumps. I transferred them to the old Zimbra, and to my surprise, the system started. However, the users in the admin panel remained the same as before, likely because I didn’t transfer LDAP. I then tried to transfer /opt/zimbra/data/ldap from the new system to the old one using rsync, but that attempt failed:
Attempts to export the LDAP database from the new broken Zimbra using zmslapcat also failed:
I have attached various logs.
Please help me get Zimbra back up and running! Or help me make the old Zimbra accept the new LDAP data. I would be very grateful for any support you can provide. Thank you so much for your help!
Here’s what we have:
•Ubuntu Server 16.04.3 LTS
•ZCS 8.7.11.GA.1854.UBUNTU16.64 (FOSS edition)
•An old (2020) snapshot with a working version of Zimbra
•A new (2025) snapshot with a broken version of Zimbra
The Problem:
At the beginning of this year, something went wrong with Zimbra, and I can't pinpoint exactly what happened. We experienced power issues in our server room, and the TLS certificate for Zimbra expired. I tried to recreate it using Certbot and acme.sh. At the same time, the Zimbra Collaboration Suite also expired. It’s possible that one of these issues, or a combination of them, caused the problem.
It all began when emails stopped being sent (only within the local network). I then discovered the expired certificate and made several unsuccessful attempts to recreate it. I also noticed that the contents of the /opt/zimbra/bin directory were missing. I ran ./install.sh to regenerate the system, but it only made things worse. Now, the system won’t start, and I get the following error:
Code:
zimbra@zimbra:~$ zmcontrol statusConnect: Unable to determine enabled services from LDAP.Unable to determine enabled services. Cache is out of date or doesn't exist.Code:
zimbra@zimbra:~$ zmprov -l gaa | xargs -I {} zmprov ga {}[] INFO: master is down, falling back to replica...[] FATAL: failed to initialize LDAP clientcom.zimbra.cs.ldap.LdapException: LDAP error: : An error occurred while attempting to connect to server localhost:389: java.io.IOException: An error occurred while attempting to establish a connection to server localhost:389: java.net.ConnectException: Connection refused (Connection refused)ExceptionId:main:1738261831655:71b2b3932a3c9b02Code:ldap.LDAP_ERRORat com.zimbra.cs.ldap.LdapException.LDAP_ERROR(LdapException.java:90)at com.zimbra.cs.ldap.unboundid.UBIDLdapException.mapToLdapException(UBIDLdapException.java:74)at com.zimbra.cs.ldap.unboundid.UBIDLdapException.mapToLdapException(UBIDLdapException.java:40)at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnPool(LdapConnectionPool.java:117)at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnectionPool(LdapConnectionPool.java:63)at com.zimbra.cs.ldap.unboundid.UBIDLdapContext.init(UBIDLdapContext.java:111)at com.zimbra.cs.ldap.unboundid.UBIDLdapClient.init(UBIDLdapClient.java:39)at com.zimbra.cs.ldap.LdapClient.getInstanceIfLDAPavailable(LdapClient.java:62)at com.zimbra.cs.ldap.LdapClient.getInstance(LdapClient.java:69)at com.zimbra.cs.ldap.LdapClient.initialize(LdapClient.java:94)at com.zimbra.cs.account.ldap.LdapProv.<init>(LdapProv.java:47)at com.zimbra.cs.account.ldap.LdapProvisioning.<init>(LdapProvisioning.java:279)at com.zimbra.cs.account.ldap.LdapProvisioning.<init>(LdapProvisioning.java:276)at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at java.lang.Class.newInstance(Class.java:442)at com.zimbra.cs.account.Provisioning.getInstance(Provisioning.java:287)at com.zimbra.cs.account.Provisioning.getInstance(Provisioning.java:244)at com.zimbra.cs.account.ProvUtil.initProvisioning(ProvUtil.java:1004)at com.zimbra.cs.account.ProvUtil.main(ProvUtil.java:3955)Caused by: LDAPException(resultCode=91 (connect error), errorMessage='An error occurred while attempting to connect to server localhost:389: java.io.IOException: An error occurred while attempting to establish a connection to server localhost:389: java.net.ConnectException: Connection refused (Connection refused)')at com.unboundid.ldap.sdk.LDAPConnection.connect(LDAPConnection.java:754)at com.unboundid.ldap.sdk.LDAPConnection.connect(LDAPConnection.java:686)at com.unboundid.ldap.sdk.LDAPConnection.<init>(LDAPConnection.java:518)at com.unboundid.ldap.sdk.SingleServerSet.getConnection(SingleServerSet.java:229)at com.unboundid.ldap.sdk.ServerSet.getConnection(ServerSet.java:98)at com.unboundid.ldap.sdk.LDAPConnectionPool.createConnection(LDAPConnectionPool.java:938)at com.unboundid.ldap.sdk.LDAPConnectionPool.<init>(LDAPConnectionPool.java:876)at com.unboundid.ldap.sdk.LDAPConnectionPool.<init>(LDAPConnectionPool.java:779)at com.unboundid.ldap.sdk.LDAPConnectionPool.<init>(LDAPConnectionPool.java:726)at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnPool(LdapConnectionPool.java:114)... 18 moreCaused by: java.io.IOException: An error occurred while attempting to establish a connection to server localhost:389: java.net.ConnectException: Connection refused (Connection refused)at com.unboundid.ldap.sdk.LDAPConnectionInternals.<init>(LDAPConnectionInternals.java:137)at com.unboundid.ldap.sdk.LDAPConnection.connect(LDAPConnection.java:744)... 27 moreAt this point, I compared the configuration of the old working Zimbra from 2020 [attachment: 2020-old-n-working-conf.txt] with the new broken Zimbra [attachment: 2025-new-n-broken-conf.txt] and found that many variables, especially those related to LDAP and MySQL (where most errors occur), differ, including passwords. I suspect this is part of the problem, but I’m unsure how to fix it. I updated the certificates, fixed permissions using zmfixperms, and even tried importing the output of zmlocalconfig -s from the old version to the new one, but I still encountered startup errors (maybe I missed something).
The broken Zimbra won’t let me change the ldap_root_password with the command (even though it seems this password is stored in plain text):
Code:
zimbra@zimbra:~$ zmldappasswd -r new_root_passUpdating local config and LDAPconnect: Connection refused at /opt/zimbra/bin/zmldappasswd line 101, line 755.Attempt to Import Databases:
At one point, I decided to stop trying to fix the broken system and instead transfer the MySQL and LDAP databases from the broken Zimbra to the old working version. Interestingly, I was able to successfully obtain the MySQL database dumps. I transferred them to the old Zimbra, and to my surprise, the system started. However, the users in the admin panel remained the same as before, likely because I didn’t transfer LDAP. I then tried to transfer /opt/zimbra/data/ldap from the new system to the old one using rsync, but that attempt failed:
Code:
zimbra@zimbra:~$ zmcontrol startHost zimbra.newinfosec.ruStarting ldap...Done.Search error: Unable to determine enabled services from ldap.Enabled services read from cache. Service list may be inaccurate.Starting zmconfigd...Failed.Starting zmconfigd...failed.Starting logger...Failed.Starting logswatch...[] INFO: master is down, falling back to replica...[] FATAL: failed to initialize LDAP clientcom.zimbra.cs.ldap.LdapException: LDAP error: : invalid credentialsExceptionId:main:1739209164041:22eb43355ffc9535Code:ldap.LDAP_ERRORat com.zimbra.cs.ldap.LdapException.LDAP_ERROR(LdapException.java:90)at com.zimbra.cs.ldap.unboundid.UBIDLdapException.mapToLdapException(UBI DLdapException.java:74)at com.zimbra.cs.ldap.unboundid.UBIDLdapException.mapToLdapException(UBI DLdapException.java:40)at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnPool(LdapCo nnectionPool.java:117)at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnectionPool( LdapConnectionPool.java:63)at com.zimbra.cs.ldap.unboundid.UBIDLdapContext.init(UBIDLdapContext.jav a:111)at com.zimbra.cs.ldap.unboundid.UBIDLdapClient.init(UBIDLdapClient.java: 39)at com.zimbra.cs.ldap.LdapClient.getInstanceIfLDAPavailable(LdapClient.j ava:62)at com.zimbra.cs.ldap.LdapClient.getInstance(LdapClient.java:69)at com.zimbra.cs.ldap.LdapClient.initialize(LdapClient.java:94)at com.zimbra.cs.account.ldap.LdapProv.<init>(LdapProv.java:47)at com.zimbra.cs.account.ldap.LdapProvisioning.<init>(LdapProvisioning.j ava:279)at com.zimbra.cs.account.ldap.LdapProvisioning.<init>(LdapProvisioning.j ava:276)at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at java.lang.Class.newInstance(Class.java:442)at com.zimbra.cs.account.Provisioning.getInstance(Provisioning.java:287)at com.zimbra.cs.account.Provisioning.getInstance(Provisioning.java:244)at com.zimbra.cs.account.ProvUtil.initProvisioning(ProvUtil.java:1004)at com.zimbra.cs.account.ProvUtil.main(ProvUtil.java:3955)Caused by: LDAPException(resultCode=49 (invalid credentials), errorMessage='inva lid credentials')at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:1894)at com.unboundid.ldap.sdk.LDAPConnectionPool.createConnection(LDAPConnec tionPool.java:988)at com.unboundid.ldap.sdk.LDAPConnectionPool.<init>(LDAPConnectionPool.j ava:876)at com.unboundid.ldap.sdk.LD18:58APConnectionPool.<init>(LDAPConnectionPool.j ava:779)at com.unboundid.ldap.sdk.LDAPConnectionPool.<init>(LDAPConnectionPool.j ava:726)at com.zimbra.cs.ldap.unboundid.LdapConnectionPool.createConnPool(LdapCo nnectionPool.java:114)... 18 morezimbra logger service is not enabled! failed.Starting mailbox...Failed.Starting memcached...Done.Starting proxy...Done.Starting amavis...Done.Starting antispam...Done.Starting antivirus...Done.Starting opendkim...Failed.opendkim: /opt/zimbra/conf/opendkim.conf: ldap://zimbra.newinfosec.ru:389/?DKIMS elector?sub?(DKIMIdentity=$d): dkimf_db_open(): Invalid credentialsFailed to start opendkim: 0Starting snmp...Done.Starting spell...Done.Starting mta...Done.Starting stats...Done.Starting service webapp...Failed.Starting zimbra webapp...Failed.Code:
root@zimbra:/home/post# su - zimbrazimbra@zimbra:~$ /opt/zimbra/libexec/zmslapcat /opt/zimbra/backup67aafe65 mdb_db_open: database "": mdb_dbi_open(/opt/zimbra/data/ldap/mdb/db/id2v) failed: MDB_NOTFOUND: No matching key/data pair found (-30798).67aafe65 backend_startup_one (type=mdb, suffix=""): bi_db_open failed! (-30798)slap_startup failedPlease help me get Zimbra back up and running! Or help me make the old Zimbra accept the new LDAP data. I would be very grateful for any support you can provide. Thank you so much for your help!
Statistics: Posted by Zimbra_BIGGEST_Fan — Tue Feb 11, 2025 3:08 pm