Databricks Auto Loader

Auto Loader tracks which files it has already read, so you can point it at a folder and process only what is new. Most of the work is choosing a few cloudFiles options and deciding how schema changes should behave.