encoders: some cleanup to prepare new encoder operation

This MR provides some cleanups from the different encoders to prepare them for the addition of the set_format operation (needed for #25152 (closed) and #25153) and then the merge of the two encoders in follow-up MR.

In particular, most of the changes are there to accomodate the fact that set_format might be allowed to fail, but they would still make sense without this constraint, and some of them are potential memory leak that were fixed along the way.

I still need to do more testing on the different encoders, but I don't have an already defined test pipeline to check every encoders.