Do you use mpm_event?
DO you still use KeepAlive for all clients event behind of proxy?
Do you see similar messages in logs?
22.214.171.124 - - [19/May/2019:10:17:44 +0000] "-" 408 - "-" "-" 126.96.36.199 188.8.131.52
184.108.40.206 - - [19/May/2019:10:17:47 +0000] "-" 408 - "-" "-" 220.127.116.11 18.104.22.168
22.214.171.124 - - [19/May/2019:10:17:48 +0000] "-" 408 - "-" "-" 126.96.36.199 188.8.131.52
Obvious better variant – disable KeppAlive at all.
Also you can find out who from external client so slow behavious – some system of health check or Slow DDOS.
Also you can see a lot of processes in dashboard of server (mod_status) with string of request – NULL. Usually it will be on default 443 port.
Are you by any chance running your web servers in Amazon behind an Elastic Load Balancer?
It seems they generate a lot of 408 responses due to their health checks.
Then this is your way:
RequestReadTimeout header=65 body=65
ELB uses very high timeout (under Connection Settings -> Idle Timeout)
Some of the solutions to set:
RequestReadTimeout header=0 body=0
This disables the 408 responses if a request times out. Change the ELB health check to a different port.
Disable logging for the ELB IP addresses with:
SetEnvIf Remote_Addr "10\.0\.0\.5" exclude_from_log
CustomLog logs/access_log common env=!exclude_from_log
Also may be other explanation in context of migration from http->https and logs in content of first defauklt SSL vhost.
I suppose that you find these log entries only in the logs of the default (or alphabetically first) apache ssl conf and
that you have a low timeout ‘less 20’. As of my tests these are clients establishing pre-connected/speculative sockets
to your web server for fast next page/resource load. Since they only establish the initial socket connection or handshake
(150 bytes or few thousands) the connect to the ip and do not specify a vhost name, and got logged in the
default/firs apache conf log.
After few secs from the initial connection they drop the socket if not needed or the use is for faster further request.
If your timeout is lower than these few secs you get the 408 if is higher apache doesn’t bother. So either you ignore them
or add a different default conf for apache, or you rise the timeout having more apache processes busy waiting from the client
to drop or use the socket. But it is nor very good.