Although the spawns are random you really should find some. Are you checking in the right place, bandits cave near the river? It actually is really hard to notice and I've passed over it a few times, so check carefully.
Eventually I'm going to edit the map to show spawns for that damn fungus...