Issue: Upgrade 8.5.9 -> 8.5.12 failed

Using Concrete / Help
1

PHP version is 7.4
No add-ons installed on concrete5.
Theme is Clonemental.

Error message: “The system ran out of semaphores. You have to free some semaphores, or disable semaphore-based mutex.”

After modifying /config/update.php back to previous version, site still worked.

Anyone have clue where is the problem? PHP/MySQL/Server/other?

Here is enviroment info if it helps

concrete5 Version

Core Version - 8.5.9
Version Installed - 8.5.9
Database Version - 20220319043123

Database Information

Version: 5.7.44-log
SQL Mode:

concrete5 Packages

Cloneamental (0.9.3)

concrete5 Overrides

None

concrete5 Cache Settings

Block Cache - Off
Overrides Cache - Off
Full Page Caching - On - If blocks on the particular page allow it.
Full Page Cache Lifetime - Every 6 hours (default setting).

Server Software

Apache

Server API

cgi-fcgi

PHP Version

7.4.33

PHP Extensions

bcmath, bz2, calendar, cgi-fcgi, Core, ctype, curl, date, dba, dom, exif, fileinfo, filter, ftp, gd, gettext, gmp, hash, http, iconv, imagick, imap, intl, ionCube Loader, json, libxml, mbstring, mysqli, mysqlnd, newrelic, odbc, openssl, pcntl, pcre, PDFlib, PDO, pdo_dblib, PDO_Firebird, pdo_mysql, PDO_OCI, PDO_ODBC, pdo_pgsql, pdo_sqlite, pdo_sqlsrv, Phar, posix, propro, pspell, raphf, readline, Reflection, session, shmop, SimpleXML, soap, sockets, SPL, sqlite3, standard, sysvmsg, sysvsem, sysvshm, tidy, tokenizer, xml, xmlreader, xmlrpc, xmlwriter, xsl, yaml, zip, zlib

PHP Settings

max_execution_time - 30
log_errors_max_len - 1024
max_file_uploads - 20
max_input_nesting_level - 64
max_input_time - 60
max_input_vars - 1000
max_multipart_body_parts - -1
memory_limit - 128M
post_max_size - 8M
upload_max_filesize - 2M
ic24.api.max_timeout - 7
mbstring.regex_retry_limit - 1000000
mbstring.regex_stack_limit - 100000
mysqli.max_links - Unlimited
mysqli.max_persistent - Unlimited
newrelic.application_logging.forwarding.max_samples_stored - 10000
newrelic.custom_events.max_samples_stored - 30000
newrelic.span_events.max_samples_stored - 2000
newrelic.special.max_nesting_level - -1
newrelic.transaction_tracer.max_segments_cli - 100000
newrelic.transaction_tracer.max_segments_web - 0
odbc.max_links - Unlimited
odbc.max_persistent - Unlimited
pcre.backtrack_limit - 1000000
pcre.recursion_limit - 100000
pdo_sqlsrv.client_buffer_max_kb_size - 10240
raphf.persistent_handle.limit - -1
session.cache_limiter - no value
session.gc_maxlifetime - 7200
soap.wsdl_cache_limit - 5
unserialize_max_depth - 4096

2

If the error message is accurate (ran out of semaphores) and not secondary, try running the update from the CLI.

The CLI does not run the full core, so if you are lucky it will not run out of semaphores.

Before updating: disable and clear the cache, put doctrine entities into development mode, put the site in maintenance mode.

Also, update to the latest 8.5.v (about 8.5.14 I think). Core 8.5.12 has been superseded.

3

Cache turned off and cleared. Doctrine switched to dev mode.
Same error with 8.5.15 also.
I have no access to CLI/SSH. So I will try to contact servers helpdesk, if they can do something about it.

Thanks!

4

I got SSH access to server, but same semaphore error occured with CLI also. :frowning:

5

I asked from my server helpdesk if there is any possibility to increase semaphores on that website.

My servers helpdesks answer: “Unfortunately, the purpose of semaphores in this context was not really clear to me either. Their number can only be increased at the server level, not per account. Therefore, a change cannot be made on very small grounds. However, we are still investigating the matter. What makes it interesting is that some of your sites are updated and some are not.”

I have about 30 Concrete5 websites on that same server and i have updated 20 of them to ConcreteCMS 9.x. A few got same semaphore issue and rest of them have some add-ons or themes witch not support v9.x.

6

If you are committed to the host, the way round this is to clone the site onto a local development system, do the update, then clone that back to replace the live site.

Its a more involved process, but much better for solving problems and minimal risk to the live site.

You could also look for differences in the environment between your problem site and a site successfully updated.

You may find Environment Diff useful. It is only tested to run on 8.5.12+, so you would install on a successfully updated site and then paste in the environment report from a problem site.

7

Another thought. If the issue is at the server level, could it be traffic on the other sites that are using up the semaphores? Maybe pick a slow time in the middle of the night and put all the other sites into maintenance mode.

Or maybe semaphores are not being released and hence the available pool is diminishing. In which case you could try rebooting the server. Something else to try in the middle of the night.

Both a bit of a long shot.

8

This must be some sort of server issue. Now I got same error updating ConcreteCMS 9.1.3 →

I can’t do anything on the server myself, so I’m trying to get the server maintenance people to figure out the problem.

9

All semaphores of that webhotel removed by server heldesk and then update works again. No reboot required.

There was 4 “reserved” semaphores before remove. After update there is still 2 semaphores in use. Is this normal or should concrete release those after update?

10

I don’t know. Perhaps @andrew can advise.

11

When you upload a new Concrete version to the webserver, and when someone visits the website right after that, Concrete starts the “upgrade” process automatically.

Problems may arise if two users visits the website at the same time: we risk to have two “upgrade” processes running in parallel, and that would probably break the website.

In order to prevent this case, Concrete uses those system semaphores: a semaphore can be “owned” by a single process only, and that’s acquired by the very first website visitor.
The second visitor trying to acquire it will fail, so the upgrade process will occur just once.

Once the upgrade process has been completed, the semaphore is “released” since Concrete doesn’t need it anymore.

If for some reason that semaphore is still “reserved” to Concrete you can “free” it.
But please be sure that:

  1. The Concrete upgrade process is completed (it usually just requires just a few seconds, so you can assume that’s ok)
  2. The semaphore is actually reserved by Concrete and not by some other system process (Concrete isn’t the only stuff that uses semaphores: a lot of other system stuff may use them).
12
  1. There is “install update” button. So update won’t start automatically.
  2. When you hit install button, Concrete should go to maintenance mode, until update is complete. This site has manually put in maintenance mode anyway.
  3. Semaphores (2) was still reserved by Concrete 15 mins after update was completed. All semaphores of that webhotel were relased before updating. (I have no access to that info on server. Helpdesk told me this.)

Does normal use of site also reserve semaphores?

13

concrete5 version 8.5 only uses semaphores during the core upgrade (it uses just one semaphore, and concrete releases it as soon as the upgrade completes).

1 Like
14

Thanks. I’ll pass this information to servers helpdesk. My problem is solved.

15

If the update breaks part way, will the semaphore timeout or are there other measures in place to ensure the semaphore is released?

1 Like