I had to upgrade an older Concrete database recently to uftmb4, and found that scripts or commands in something like PHPMyAdmin to update all tables would fail, due to keys conflicting or other issues.
In the end what I did was perform a database dump, then open exported SQL in a text editor and do some find and replaces on the the CHARSET and COLLATE values. It didn’t take long to do, but I just had to carefully perform the find and replaces so that it would handle all occurrences.
I then wiped my database and re-imported it via my updated database dump. That worked fine.
Doing it this way I don’t think would be considered good database practice in general, but for this specific case of going from utf8 to utf8mb4 I believe it’s going to be pretty safe to do.