My first question is, where is this guy getting AV1 videos? Never seen these on the high seas.
Also, given that these videos are going to be reencoded, which is tremendously expensive, I feel that any optimization in this step is basically premature. Naively launching ffprobe 10,000 times is probably still less heavyweight than 1 reencode.
Right click on a YouTube video and select "Stats for Nerds" to see which format it's using in your browser. AV1 will be something like "av01.0.09M.08".
You've probably watched a lot of AV1 video without realising it.
I exclusively download av1 encodes from places like tbp. It has fantastic quality for the filesize, and AV1 also benefits the most from the trick of encoding sdr content in 10 bit (more accurate quantization at a smaller size). Crazy that we can fit ~two hours of 1080p video at better than netflix quality (they bias their psnr/etc a little low for my eyes) on a single CD.
I'm not sure it's fair to call reencodes expensive. Sure, its relatively expensive to using ffprobe, but any 4 series nvidia gpu with 2 nvenc engines can handle five? simultaneous realtime encodes, or will get up to near 180fps if it isn't being streamed. Our "we have aja at home" box with four of them churned through something like 20,000 hours of video in just under two weeks.
My understanding is that you shouldn't be using HW accelerated encoding for any archival purpose except realtime capture.
The PSNR/bitrate is much lower for HW encode, but the encode rate is typically realtime or better. That's a great tradeoff if you are transcoding so that a device with limited bandwidth can receive the video while streaming, or so that you can encode a raw livestream from a video capture or camera. It's not so great if you are saving to disk and planning to watch multiple times.
Sounds like you're just sailing the wrong seas. Some have plenty of AV1. Though those tend to be more obviously advertised as such, I believe, so perhaps this is about downloads from YouTube.
Also, given that these videos are going to be reencoded, which is tremendously expensive, I feel that any optimization in this step is basically premature. Naively launching ffprobe 10,000 times is probably still less heavyweight than 1 reencode.