I have 5 pipelines scheduled to run every minute with a max concurrent run of 1 for each pipeline. Each run provisions a cluster and then runs the pipeline.
At any point in time, I don't expect more than 5 clusters to be provisioned due to the max concurrent runs constraint. However, the max concurrent runs only considers RUNNING state, whereas a program will be in PENDING state during provisioning. This leads to multiple active runs for a given pipeline that has a max concurrent run of 1. In this particular case, there were more than 20 clusters provisioned at a given time.