Y2k22 bug is causing Microsoft Exchange Server to fail worldwide: FIP-FS Scan Engine failed to load

Company admins are having their New Year’s celebrations interrupted by reports that their Exchange Servers are failing with the error “FIP-FS Scan Engine failed to load – Can’t Convert “2201010001” to long (2022/01/01 00:00 UTC)“.

The issue appears to be due to Microsoft using the first two numbers of the updated version to denote the year of the update, which caused the “long” version of the date to overflow.

At present, it seems the primary workaround is to disable the anti-malware scanner on the Exchange Server by using Set-MalwareFilteringServer -BypassFiltering $True -identity <server name> and restarting the Microsoft Exchange Transport service.

It appears Microsoft has not acknowledged the issue yet, but if you are affected, some peer support is available at Reddit here.

Update: Microsoft has now acknowledged the issue and is working on a fix. They write:

We are aware of and working on an issue causing messages to be stuck in transport queues on Exchange Server 2016 and Exchange Server 2019. The problem relates to a date check failure with the change of the new year and it not a failure of the AV engine itself. This is not an issue with malware scanning or the malware engine, and it is not a security-related issue. The version checking performed against the signature file is causing the malware engine to crash, resulting in messages being stuck in transport queues.

On the coming fixes, they say:

Our engineers were working around the clock on a fix that would eliminate the need for customer action, but we determined that any change that did not involve customer action would require several days to develop and deploy. We are working on another update which is in final test validation. The update requires customer action, but it will provide the quickest time to resolution.

In the meantime, Microsoft says if you have other ways of sanitizing your email, such as routing them through exchange online, you can use the following methods to disable the malware scanner:

• Antimalware protection in Exchange Server | Microsoft Docs
• Procedures for anti-malware protection in Exchange Server | Microsoft Docs

Update 2:  An official fix is now available from Microsoft, which involved stopping and updating the Microsoft Filtering Management service and updating the engine to the latest version.

Remove existing engine and metadata
1. Stop the Microsoft Filtering Management service.  When prompted to also stop the Microsoft Exchange Transport service, click Yes.
2. Use Task Manager to ensure that updateservice.exe is not running.
3. Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
4. Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.

Update to latest engine
1. Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
2. Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.

Verify engine update info
1. In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
2. Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001.

After updating the engine, we also recommend that you verify that mail flow is working and that FIPFS error events are not present in the Application event log.

Microsoft has also released a script at  https://aka.ms/ResetScanEngineVersion that automates the process and runs parallel over all your servers.

Read all the associated detail at Microsoft here.

Thanks, Ola, for the tip.