The lyric “Blow the man down” may refer to the act of knocking a man to the ground. (source)
Come all you young fellows who follow the sea,
To me way! Hey! Blow the man down.
Now pray, pay attention and listen to me,
Give me some time to blow the man down.
Song pdf & midi below