To either improve one's behavior or else be required to leave; to either improve one's performance in an activity or else withdraw from that activity completely
Portions of those letters quoted by Bruccoli indicate that though Hemingway could be sympathetic, he used a lot of ink telling Fitzgerald to shape up or ship out.
If something is shipshape, it looks tidy, neat, and in good condition. The house only needs an occasional coat of paint to keep it shipshape. neat and clean (shipshapen (1600-1700), from shapen, old past participle of shape)