Because this waterfalls exist for long time with a mysterious atmosphere, poeple made many legend such as Dragon, or Kappa. This waterfalls was enshrined Benzaiten. So, people used pray for raining.
In 1844, people built Onsen because there was hot spring around this waterfalls. That is called Fujimi Onsen, and today, its Onsen still exist next to this falls.