I also worry about the fines settling in to your shop and stirring up later. Later you are doing something you don't think you need a mask for and inhaling it.
That's valid. I'm in the middle of configuring a new shop so perhaps I should rethink the collector location. I do run the air filtration system 100% of the time I'm in the shop and for 12 hours after I leave it which turns to air over every 3-4 minutes. I only run the chip separator and Hepa filter when I'm cutting chips or sanding.