The Secretary of the Interior shall reserve and may exercise the authority to cancel any prospecting permit upon failure by the permittee to exercise due diligence in the prosecution of the prospecting work in accordance with the terms and conditions stated in the permit, and shall insert in every such permit issued under the provisions of this chapter appropriate provisions for its cancellation by him.