When I mention that I would try to optimize this, I was more thinking about writing a dedicated script just to do that.

But this here could also be interesting.

I'm not sure thought that I would put this into RETURNN yet, while you are still experimenting with it. You can rather put this somewhere into your i6_experiments.users. and then just use it from there. (I think we might need to extend the class resolution slightly, if "." in cls_name: ...; see similar code in rf.build_from_dict or get_optimizer_class or other such functions.)

This is a first attempt at automatically optimizing the bucket limits during training. After every subepoch the limits are adjusted to make every bucket catch a roughly equal number of segments.

With "limit", you mean the max num seqs of a bucket?

I still don't really understand why this is something you want to optimize for. Why does it matter that they are evenly distributed?

This is to get better speed or better model performance or both? I don't see how this affects speed. I also don't see/understand how this affects model performance.

I thought that optimizing the max seq lens of each bucket, i.e. the boundaries, would be a more reasonable thing to optimize w.r.t. minimizing the amount of padding. That would improve at least the speed.

It seems this PR includes all the epoch_continuous changes? Why? Are those in any way related here? I don't see how. If they are not related, can you remove them from this PR?

I don‘t think this PR is going to be merged in the current form.