6.1.7 and 6.2.6 Updates Cause Critical Error

[mping001]

Today we had around 20+ sites go down sporadically across multiple servers and hosting providers. All of the sites were on 6.1.7 or 6.2.6 and had the same errors:
PHP Fatal error: Uncaught TypeError: call_user_func_array(): Argument #1 ($callback) must be a valid callback, function “_wp_footnotes_kses_init” not found or invalid function name in /wp-includes/class-wp-hook.php:308

After verifying checksums on core we found that the wp-includes/default-filters.php file did not match and we had to re-download core on all the sites to get rid of the critical error. I imagine this will be happening for us most of the day. We checked server and security logs and didn’t find anything correlated to the issue and believe it may be something to do with the update methods. Not sure but some insight here would be helpful.

[mping001]

Here is the stack trace:
[24-Jun-2024 21:05:49 UTC] PHP Fatal error: Uncaught TypeError: call_user_func_array(): Argument #1 ($callback) must be a valid callback, function "_wp_footnotes_kses_init" not found or invalid function name in /wp-includes/class-wp-hook.php:308
Stack trace:

#0 /wp-includes/class-wp-hook.php(332): WP_Hook->apply_filters(NULL, Array)
#1 /wp-includes/plugin.php(517): WP_Hook->do_action(Array)
#2 /wp-includes/pluggable.php(48): do_action('set_current_use...')
#3 /wp-includes/user.php(3606): wp_set_current_user(0)
#4 /wp-includes/pluggable.php(70): _wp_get_current_user()
#5 /wp-content/plugins/cleantalk-spam-protect/cleantalk.php(2877): wp_get_current_user()
#6 /wp-content/plugins/cleantalk-spam-protect/cleantalk.php(2856): apbct_is_user_role_in(Array, NULL)
#7 /wp-content/plugins/cleantalk-spam-protect/inc/cleantalk-public.php(363): apbct_is_user_enable()
#8 /wp-includes/class-wp-hook.php(308): apbct_init('')
#9 /wp-includes/class-wp-hook.php(332): WP_Hook->apply_filters(NULL, Array)
#10 /wp-includes/plugin.php(517): WP_Hook->do_action(Array)
#11 /wp-settings.php(480): do_action('plugins_loaded')
#12 /wp-config.php(88): require_once('/home/cclawhg/p...')
#13 /wp-load.php(50): require_once('/home/cclawhg/p...')
#14 /wp-blog-header.php(13): require_once('/home/cclawhg/p...')
#15 /index.php(17): require('/home/cclawhg/p...')
#16 {main}
  thrown in /wp-includes/class-wp-hook.php on line 308

[peterwilsoncc]

@mping001 Thank you for the ticket, are you able to let me know the following:

• Were the sites auto-updated or manually updated
• If manually updated, was this done by downloading the update from wordpress.org, via wp-cli or another method?

[mping001]

Replying to peterwilsoncc:

@mping001 Thank you for the ticket, are you able to let me know the following:

• Were the sites auto-updated or manually updated
• If manually updated, was this done by downloading the update from wordpress.org, via wp-cli or another method?

It was automatically updated. We have minor updates enabled in the wp-config file. But when the sites went down with the critical errors we downloaded using: wp core update --force --version='6.1.7' or '6.2.6' then it did verify checksums and the fatal error was gone.
Edit because I failed to mention that the issue came back a few times for a couple of the sites even after updating via cli.

[narenin]

Hi @mping001

The error stack trace indicates an issue with the CleanTalk plugin for WordPress. Specifically, it is caused by a function _wp_footnotes_kses_init that is either missing or incorrectly named. This function is being called via call_user_func_array in the class-wp-hook.php file, which handles WordPress hooks and actions.

Could you please try to deactivate plugins or check for the updates.

[thismatters]

Same here. After auto-update 9 of 12 sites are offline. Needed to upload the version before data. Time for me to disable Auto Update Function for Security Updates to.

Version: 6.5.5 and 6.4.5 and 6.2.5

[mping001]

@narenin I first thought that as well. We updated cleantalk, and we looked into the _wp_footnotes_init error and it appears that should only be used by core wordpress. We verified checksums on the current update and they did not match after the automatic update and the default-filters.php file failed the checksum which we believe was the source of the issue. Upon re-downloading the update file using wp-cli the site is functioning normally.

[sberney]

Same here with 1 site that automatically upgrade to v6.2.6.

We needed to update this site to 6.5.5 to get rid of the error.

[peterwilsoncc]

@thismatters @sberney Were each of you seeing the same errors regarding _wp_footnotes_kses_init?

[sberney]

Replying to peterwilsoncc:

@thismatters @sberney Were each of you seeing the same errors regarding _wp_footnotes_kses_init?

Yes !

[thismatters]

didn`t had a look there - sorry. But rollback to the older WP Version solves this issue. (Uploading wp-include and wp-admin folders).

[thim00]

Replying to peterwilsoncc:

@thismatters @sberney Were each of you seeing the same errors regarding _wp_footnotes_kses_init?

We have a site that has just been updated from version 6.2.5 to 6.2.6 (while our staging environment remains at 6.2.5 and is functioning normally), and we are also seeing this error with the same function _wp_footnotes_kses_init. Several plugins are affected by this missing function, such as GiveWP.

Edit : Rollback production to version 6.2.5 fixed it.

[jorbin]

First off, a reminder that only the most recent in the 6.5 series is safe to use and actively maintained. While security patches are backported as a courtesy, you are strongly encouraged to only use the actively maintained version.

I've checked the following zips so far:

• ​https://downloads.wordpress.org/release/wordpress-6.2.6-partial-0.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6-partial-1.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6-partial-2.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6-partial-3.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6-partial-4.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6-partial-5.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6-no-content.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6-new-bundled.zip

None of them contain a file with any reference to _wp_footnotes_kses_init.

Does anyone have logs on their server to indicate which zip file they received that contains the invalid file?

I know that some hosts will proxy requests and supply their own zips, so if possible could you also reach out to your host if you are experiencing this and share this ticket and ask them if they have any additional information. I'm also reaching out to the meta team to ask them to check all of the files on the download server for any I may have missed so we can see if one of them has an incorrect file.

[sberney]

I've just noticed that when commenting on lines 602-603 of the file "/wp-includes/default-filters.php" :

// Footnotes Block.
//add_action( 'init', '_wp_footnotes_kses_init' ); //comment to get rid of error
//add_action( 'set_current_user', '_wp_footnotes_kses_init' ); //comment to get rid of error

makes the error disappear

But I don't have any logs of the file that was downloaded during update.

Maybe it's related to the add_action/filter command that didn't load the file containing the "_wp_footnotes_kses_init" function before calling it here ?!

[cdsaenz]

Replying to sberney:

I've just noticed that when commenting on lines 602-603 of the file "/wp-includes/default-filters.php" :

// Footnotes Block.
//add_action( 'init', '_wp_footnotes_kses_init' ); //comment to get rid of error
//add_action( 'set_current_user', '_wp_footnotes_kses_init' ); //comment to get rid of error

makes the error disappear

But I don't have any logs of the file that was downloaded during update.

Maybe it's related to the add_action/filter command that didn't load the file containing the "_wp_footnotes_kses_init" function before calling it here ?!

Thanks @sberney this was it. I just downloaded the current 6.2.6 release zip and replaced default-filters.php in all my sites, now they're working fine.

[roytanck]

Looks like _wp_footnotes_kses_init was introduced in WP 6.3.2, and is therefore not available in prior versions.

​https://developer.wordpress.org/reference/functions/_wp_footnotes_kses_init/#changelog

[mping001]

Replying to jorbin:

First off, a reminder that only the most recent in the 6.5 series is safe to use and actively maintained. While security patches are backported as a courtesy, you are strongly encouraged to only use the actively maintained version.

I've checked the following zips so far:

• ​https://downloads.wordpress.org/release/wordpress-6.2.6-partial-0.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6-partial-1.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6-partial-2.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6-partial-3.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6-partial-4.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6-partial-5.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6-no-content.zip
• ​https://downloads.wordpress.org/release/wordpress-6.2.6-new-bundled.zip

None of them contain a file with any reference to _wp_footnotes_kses_init.

Does anyone have logs on their server to indicate which zip file they received that contains the invalid file?

I know that some hosts will proxy requests and supply their own zips, so if possible could you also reach out to your host if you are experiencing this and share this ticket and ask them if they have any additional information. I'm also reaching out to the meta team to ask them to check all of the files on the download server for any I may have missed so we can see if one of them has an incorrect file.

I don't think the zips have an incorrect file. The automated update doesn't include the same default-filters.php file that is included when updating the files manually. The automatic update files do not pass checksum verification in wp-cli which means they have been altered or are not included in that release. When you manually download the files the proper file is included and the fatal errors go away.

[arnaudj]

I share with you some hacks to get rid of this error.

First, is to touch WP core code, commenting some lines in /wp-includes/default-filters.php (as mentionned before) or add a short code in wp-load.php :

<?php
// QUICKFIX https://core.trac.wordpress.org/ticket/61489
function _wp_footnotes_kses_init(){ return false; }

Theses changes will be overwritten by the next WP update.

Second, rollback to previous version if you have wp-cli.org tool installed in the local/remote terminal:

wp core update --version=6.2.5 --force

wp core update --version=6.1.6 --force

[jorbin]

The automated update doesn't include the same default-filters.php file that is included when updating the files manually

The automated updates use those zips which is why I'm investigating those.

Walking through the default-filters file from 6.3, 6.4. and 6.5, none of them have the Footnotes Block code on line 602 so I'm not sure where this file is coming from at all. Would someone be able to upload the file they are seeing on a broken site?

[sberney]

Here's the file (dated on our server from 17.10.2023) : ​https://databird.ch/_dev/bug_wp/default-filters.zip

[tomsommer]

Some debug information, as requested on Slack:

[example.com@server ~]$ fgrep -R '_wp_footnotes_kses_init' *
public_html/wp-includes/default-filters.php:add_action( 'init', '_wp_footnotes_kses_init' );
public_html/wp-includes/default-filters.php:add_action( 'set_current_user', '_wp_footnotes_kses_init' );
^C
[example.com@server ~]$ cd public_html/
[example.com@server public_html]$ wp version
PHP Warning:  call_user_func_array() expects parameter 1 to be a valid callback, function '_wp_footnotes_kses_init' not found or invalid function name in /var/www/example.com/public_html/wp-includes/class-wp-hook.php on line 307
Warning: call_user_func_array() expects parameter 1 to be a valid callback, function '_wp_footnotes_kses_init' not found or invalid function name in /var/www/example.com/public_html/wp-includes/class-wp-hook.php on line 307
Error: 'version' is not a registered wp command. See 'wp help' for available commands.
[example.com@server public_html]$ wp core version
6.0.9
[example.com@server public_html]$ ls -l wp-includes/default-filters.php 
-rw-r--r-- 1 example.com example.com 31071 Apr 11 04:41 wp-includes/default-filters.php

[mping001]

default-filters.php file from a 6.0.9 update that broke

[mping001]

default-filters.php from 6.0.9 update after re-downloading it with wp-cli

[mping001]

Replying to jorbin:

The automated update doesn't include the same default-filters.php file that is included when updating the files manually

The automated updates use those zips which is why I'm investigating those.

Walking through the default-filters file from 6.3, 6.4. and 6.5, none of them have the Footnotes Block code on line 602 so I'm not sure where this file is coming from at all. Would someone be able to upload the file they are seeing on a broken site?

Ah sorry. We just had it happen to a site with 6.0.9 and it didn't take the site down but put the error on the homepage for some reason.
I uploaded the bad file here: https://core.trac.wordpress.org/attachment/ticket/61489/default-filters-error.php
and good file here: https://core.trac.wordpress.org/attachment/ticket/61489/default-filters-good.php

If I get another 6.1.7 or 6.2.6 I will add as well.

[mping001]

Replying to mping001:

Replying to jorbin:

The automated update doesn't include the same default-filters.php file that is included when updating the files manually

The automated updates use those zips which is why I'm investigating those.

Walking through the default-filters file from 6.3, 6.4. and 6.5, none of them have the Footnotes Block code on line 602 so I'm not sure where this file is coming from at all. Would someone be able to upload the file they are seeing on a broken site?

Ah sorry. We just had it happen to a site with 6.0.9 and it didn't take the site down but put the error on the homepage for some reason.
I uploaded the bad file here: https://core.trac.wordpress.org/attachment/ticket/61489/default-filters-error.php
and good file here: https://core.trac.wordpress.org/attachment/ticket/61489/default-filters-good.php

If I get another 6.1.7 or 6.2.6 I will add as well.

We have vulnerability patching detection that is triggering on the default-filters.php with the following details:

Name:
XSS vulnerability in the footnotes block
Description:
The footnotes block is not adequately protected against an XSS attack.

This was not detected prior to the updates and we have patching disabled. We keep having the issue come back on sites that we previously fixed and have no idea why the file is being reverted. Nothing in the logs points to anything replacing the file.

[mping001]

default-filters.php file from version 6.2.6 that causes critical error

[peterwilsoncc]

@mping001 @tomsommer @sberney @thismatters Could one of you provide the URL of an affected site? If you don't wish to share it on a public ticket, feel free to reach out to me via DM in wordpress.slack.com

[dd32]

Just to confirm, are the sites running with major-version auto-updates disabled?

ie. What do you see on wp-admin/update-core.php?

This site is automatically kept up to date with maintenance and security releases of WordPress only.
Enable automatic updates for all new versions of WordPress.

or

This site is automatically kept up to date with each new version of WordPress.
Switch to automatic updates for maintenance and security releases only.

I'm assuming the former, but I'm pondering if this is caused by a failed major-version upgrade.

Additionally; Does the site have any local-package set? Look for $wp_local_package = ... in wp-includes/version.php.

[sberney]

Replying to dd32:

On my side, it's on maintenance&security only :

"This site is automatically kept up to date with maintenance and security releases of WordPress only"

And in wp-includes/version.php, we have $wp_local_package = 'fr_FR';

[rejnok2]

site - rehab ilitace org (now fixed by quickfix from @arnaudj to wp-load.php that solve olny public site, but wp-admin still error)
error-default-filters.php file: ​https://wetransfer.com/downloads/5a83292cebe50d1425706b330e1c084b20240626114828/4cbe39e3c3e5cacb76eb5ea7e883daf820240626114828/107444?trk=TRN_TDL_01&utm_campaign=TRN_TDL_01&utm_medium=email&utm_source=sendgrid

[greenlightsolutions]

For one site that was affected by this problem, I looked in a backup of an old version of the site.
In that version (v6.2.3), there was a definition of the function _wp_footnotes_kses_init() in the file wp-includes/blocks.php.

So it may be the case that the problem is not that a reference to that function was added in wp-includes/default-filters.php, but that the definition of that function was removed from wp-includes/blocks.php.

That particular site had no wp_local_package line in wp-includes/version.php, and is set to be "automatically kept up to date with maintenance and security releases of WordPress only".

[tomsommer]

It looks like some kind of patching-software might have modified the core files, and now that blocks.php has been reverted due to an update this breaks that modification and thus the site.

[sabernhardt]

Please avoid editing wp-load or other core files, even for a quick fix. I reproduced the errors by replacing the default-filters file in my local 6.2 installation with the file from comment:25, and then I ​removed the Footnotes block actions in a custom plugin (as ​must-use so it did not need activation).

​Support forum topics suggested other quick fixes, too, and the ​WordPress.com forum topic mentions re-downloading core to correct the issue.

[sabernhardt]

For a possible patch, I could suggest checking that the function exists before adding the actions.

if ( function_exists( '_wp_footnotes_kses_init' ) ) {
	add_action( 'init', '_wp_footnotes_kses_init' );
	add_action( 'set_current_user', '_wp_footnotes_kses_init' );
}

Without knowing how those hooks were added to the default-filters file in older versions, however, I am not highly confident that any site trying to call the function in some way would get that update too.

[dd32]

Replying to greenlightsolutions:

For one site that was affected by this problem, I looked in a backup of an old version of the site.
In that version (v6.2.3), there was a definition of the function _wp_footnotes_kses_init() in the file wp-includes/blocks.php.

The function has only existed in WordPress 6.3+, introduced to the branch in [56848].

Replying to tomsommer:

It looks like some kind of patching-software might have modified the core files, and now that blocks.php has been reverted due to an update this breaks that modification and thus the site.

This is my best guess as well. Possibly a host-level security patcher. However, that doesn't make much sense to me..

The reason it doesn't make much sense, is that in the event that a core file is modified, WordPress doesn't use the "partial" zip files and instead uses the full release ZIP for that version. That would overwrite any modified files.

My best guess is that the patching software has run immediately following the upgrade. That would explain it I guess.

I'd be curious on the timing of the upgrade + fatals experienced, was it the same second, or a few minutes following, etc.

Having reviewed the logs for post-update notifications from core (WordPress sends some statistical data, and by knowing the URL of an affected site I can infer which logs are from which site, as the user-agent forms part of a unique key hash), the sites mostly used partial zips to upgrade, and appears to have run successfully. It appears that at least one probably used a full ZIP (not a partial) as it took 10x longer to upgrade than the rest, but also didn't report any issues.

[tomsommer]

Replying to dd32:

This is my best guess as well. Possibly a host-level security patcher. However, that doesn't make much sense to me..

The reason it doesn't make much sense, is that in the event that a core file is modified, WordPress doesn't use the "partial" zip files and instead uses the full release ZIP for that version. That would overwrite any modified files.

I see, okay - well then none of this makes any sense, if all files are overwritten on upgrade then the timestamp of default-filters.php should be recent.

My best guess is that the patching software has run immediately following the upgrade. That would explain it I guess.

Doubt that, based on mtime timestamps the modifications are often years old.

[markowitch]

I have an open support ticket at WordPress.org on this issue.
​https://wordpress.org/support/topic/site-not-working-after-auto-update/

[markowitch]

I found the root cause for this bug affecting the upgrade to 6.1.7 from 6.1.6.
​https://wordpress.org/support/topic/site-not-working-after-auto-update/#post-17855455

[benoitchantre]

We have some client websites that were concerned by this issue. I have found out that bad a patch was applied by Patchman. On the website I looked at (version 6.2.6), the following lines were added, but _wp_footnotes_kses_init only exist since version 6.3.2

// Footnotes Block.
add_action( 'init', '_wp_footnotes_kses_init' );
add_action( 'set_current_user', '_wp_footnotes_kses_init' );
add_filter( 'force_filtered_html_on_import', '_wp_footnotes_force_filtered_html_on_import_filter', 999 );

[jelmerverkleij]

Replying to benoitchantre:

We have some client websites that were concerned by this issue. I have found out that bad a patch was applied by Patchman. On the website I looked at (version 6.2.6), the following lines were added, but _wp_footnotes_kses_init only exist since version 6.3.2

My name is Jelmer Verkleij, I am the Director of Engineering for Patchman and have been investigating problems reported to us related to these automated patches for the last week. Unfortunately I can't find a report you made in our support tracker, so I am going off limited information for your specific case.

We have not been able to identify any Patchman-triggered patch introducing such changes yet. Can you please contact the main Patchman support channel (​https://cloudlinux.zendesk.com/hc/en-us/requests/new) with details of which website this concerns so we can debug the issue with you? If we can indeed trace this issue you report to a Patchman patch, we will of course retract and revert that patch immediately.

[jelmerverkleij]

@benoitchantre With the help of another customer who provided us debugging information, the root cause was identified.

The situation here is that this patch is part of a set of patches applied to older versions of WordPress to address an XSS vulnerability in the footnotes block. This patch was released alongside one that introduces the referenced function in blocks.php. The patches were released on 19 October 2023 and never caused any problems.

However, with the release of these newer patch versions, the default-filters.php was left untouched while blocks.php was overwritten. This caused the reference to remain but the referenced function to disappear. The same would happen in case Patchman newly detected files in this new version with partial matching: we match by file contents, not by installation version (this has proven to be significantly less error-prone in the past) and because of this, the file default-filters.php matching our expected contents for 6.1 - 6.1.6 means the patch was applicable by extrapolation.

Patchman as a product functions because updates are almost universally executed with full file overwrites, and the fixes we introduce are based on official updates towards the version that website owners usually update to. Partial updates generally involve all the files we touch. This particular flow is different because:
a) this is a partial update
b) the update in question didn't in fact cover a vulnerability that was previously considered a vulnerability by the WordPress team in a different release
c) the update was introduced long after our patches were developed and tested (and proven to be safe with every WordPress version released at that stage), thus circumventing our standard QA procedure

The odd set of circumstances has also meant that our debugging flow had to start from scratch, removing every single experience we've had with patching websites in well over a decade, in order to find out how our changes relate to the problems seen. Unfortunately, that has slowed down our ability to respond accurately to your very real problem of websites being broken. However, with help provided to us through various channels, we were able to piece together the order of events and - most importantly - what steps we can take to remediate this.

We are currently working on modifications to our vulnerability patches that will address this particular problem in a way that is safe for every installation, regardless of its upgrade path, and expect to publish those through our software within the next several hours. In order to get this executed as quickly as possible upon that release, please refer to the way your hosting provider has configured Patchman for your website, as this can be configured as your provider sees fit.

We shall also work on monitoring these partial updates as they are released in the future to better respond in cases where such partial rollouts cause problems as this one does.

For any specific questions about particular cases, please feel free to refer to our customer support portal as referenced in my previous post.

[benoitchantre]

@jelmerverkleij Thank you for your feedbacks, detailed analysis and clear communication. I understand that you have got enough information to provide a solution. If not, I have some websites that were affected by this issue that I have not yet worked on. Let me know if you need more information. Before posting here, I sent a detailed report to my host (Infomaniak).

[peterwilsoncc]

@jelmerverkleij Thank you for following up and helping the WordPress team and folks on this ticket work out the issue.

Additionally, thank you to all the people affected for your time as the root cause was worked out. I know how stressful it is when sites go down, especially in an agency context and it's happening to multiple sites you manage. Being generous with your time after the event is a credit to you all.

As it appears the bug was from another service provider, I'm going to close this ticket as works for me to indicate that the WordPress packages worked as expected.

Closing a ticket doesn't lock if from further comments so if you need to discuss this issue further, feel free to do so.

Thanks again, everyone.

[jelmerverkleij]

The amended patches in Patchman went live to our customers roughly 22 hours ago, and we have closely monitored the rollout. All cases reported to us have now been confirmed to us as fixed. If anyone is continuing to have problems, please check with your hosting provider (or through your hosting panel) whether the newest patches have been applied yet - this is dependent on how your provider has configured the automation in our product.

When in doubt, feel free to reach out directly to the Patchman customer support department so we can handle your case outside of this tracker.

Once again, I would like to apologise to all website owners affected for this chain of events; our goal is always to perform our function without affecting functionality of websites, and this novel flow made this a challenging new case for us. This is clearly something that we should amend our processes for to prevent this from happening in the future, and we always aim to improve.