I use diluted muriatic acid for scale, and the sodium carbonate electrolysis method for rust. But I have used white (distilled) vinegar and salt before for scale and it just takes longer -- usually overnight for newly purchased material wth thick scale. I do not dilute the vinegar, and just dump in some salt -- no problem if it doesn't all dissolve, you just have a saturated solution -- but I'm sure less will work fine, too. Not critical proportions, I believe.
Andrew likes citric acid. They all work.
Dilute sulfuric probably does too, but personally I feel more comfortable with dilute hardware store muriatic (hydrochloric) for some reason.
re. muriatic: I pour a little from the masonry cleaner bottle into about ten times as much water. I save the dilute stuff in a wide mouth sealed covered plastic container suitably decorated with skull and crossbones and warnings, and use it over and over again until exhausted. I keep it away from machinery and out of reach in the shop, and I only use it outdoors. It will also strip hot dip galvanizing off in a matter of minutes, if you need clean metal for welding or brazing. Does that quit vigorously.
Pekka, re. your rail material that is hard. I also had a piece of what looked like rail from a junkyard, but it actually turned out to be cast iron -- easily shattered with a sledge hammer, and though I'd intended it for a homemade anvil, I decided it wasn't suitable for that, and have found that it is fine material for melting when doing iron casting (last summer).
So. not all rail is steel, though this stuff might not have been "real" rail.