`clean` is not respected on `format.outDir`

[dmurvihill]

Reproduction link or steps

import { defineConfig } from "tsdown";

export default defineConfig({
  clean: true,
  format: {
    "esm": {
      outDir: 'esm/',
    },
    "cjs": {
      outDir: 'cjs/',
    },
  }
});

What is expected?

Put some junk in the esm/ and cjs/ folders and run tsdown. It should be cleaned up.

What is actually happening?

It doesn't get cleaned up.

Any additional comments?

No response