I have a similar situation and I left enough room between the fence and track so I could walk through. I figured that someday the fence might need to be worked on and standing on the track wasn’t going to be an option. Leaving such a wide gap has given me access to the back part of the RR for maintenance and as Steve mentioned, to fix derailments.
In some spots I used cement blocks as a retaining wall. There is one spot where I once tried hardibacker board. It was cheap and thin thus giving me more room between fence and track but within 2 years it failed due to moisture making it brittle. I replaced it with bits of plywood from another project and stakes driven into the ground like pilings which is still holding.
I think more space is better to allow air to circulate so the bottom of the fence does not rot. Greg E. has good advice.