Richard Marles has denied claims Australia’s defence will be undermined by its decision to repurpose a historical military base near Sydney’s CBD for public use.

The Albanese government announced on Sunday four heritage-listed defence sites, including Sydney’s Victoria Barracks, will remain in public hands after the plan generated considerable public backlash against their sale.

Critics, including veterans and high-profile politicians such as former prime ministers John Howard and Paul Keating, fiercely opposed the decision, with some political opponents questioning if the country’s largest city would be more vulnerable to foreign military and terrorist attacks.

The defence minister said the buildings were mostly used by office workers and no one was “seriously” suggesting the convict-labour built Victoria Barracks was fundamental to Australia’s defence.

“People making the argument know that.…