Auto Loader or COPY INTO? How to ingest files into Databricks without breaking things

A job that grew from five minutes to forty, and the one property that fixes it: knowing which files you already read