Max,
I’ll agree with the poobahs: crusher fines. However, I’ll add a couple more cents, for what they’re worth.
In a nutshell, the traditional ballast trapezoidal cross section needs serious help in a scale model outdoors application.
Since your gravel is already somewhat fine, you might try a bag of sand in one section to see if it holds out better or worse than the CF’s.
In the foreground, your gravel drops steeply off onto the landscape fabric. In my experience, that fabric tends to be slippery enough to let the gravel slide fairly easily – especially in heavy rain! I’ve had 4" thick 3/8" gravel totally wash out on that fabric.
So, you might consider holding that ballast back by using ridges in the soil, or spreading a lot more of it on additional adjacent fabric to make a flatter expanse of gravel.
Having said that, those flat rocks alongside the ballast should be doing that job fine, right?
Cliff