What does "by golly" mean ?

As everyone else has said, it's a harmless way of saying "By God" without offending anyone. These days it sounds rather old fashioned, so it's perfectly correct of Leonard H to quote examples from his grandfather.