While I initially thought about using varnish as intermediate cache to not load httpd itself, I had a quick look at current load and kind of IP blocks that spammers were using.
I've currently mitigated this (so no varnish cache, but something to still consider eventually) with two things:

• Adding https://iplists.firehol.org/?ipset=firehol_abusers_30d list to the current ipset list we have (through iptables roles)
• Added mod_qos on httpd itself to mitigate flood if ip isn't listed (yet) in DenyList listed above

Closing for now has it already had a huge perf impact while keep service running and back to normal load (no more OOM for httpd processes)