By the time of Augustus, Roman men were appearing regularly in public without their togas. Augustus tried to revive the traditional Roman dress and to enforce the wearing of a to ga in and around the Forum:

Augustus set himself to revive the ancient Roman dress and once, on seeing a group of men in dark cloaks among the crowd, quoted Virgil indignantly: 'Behold them, conquerors of the world, all clad in Roman gowns!' and instructed the aediles that no one should ever again be admitted to the Forum, or its environs, unless he wore a gown (toga) and no cloak.

Source: Gaius Suetonius Tranquillus, De uita Caesarum (Augustus 40), translated by Robert Graves (London: Penguin, 1957).