I'm not having a problem defeating him just doing it 5 times.
The first time i fought him i saved in between fights, After i defeated him the 4th time the summoner left and cave and the music was back to normal and the fiends reappeared.
I re-entered the cave and at the end of the cave i saw i had to fight him again. This time i touched the save point to heal but not save between each fight and again, after the 4th defeat the summoner just left the cave the music and fiends return and i can fight Yojimbo at the end of the cave again.
Is there a reason the 5th fight isn't happening? Am i not supposed to touch the save point and heal at all?

I did miss one of the fights, What i was doing was running straight through the cave, using your advice i found the other fight, Thank you very much.