Foul. If it's your opponent's chalk, I guess you could have an earnest discussion about who the foul was on.
I would argue the foul is still on the shooter as the rules don’t specify who’s chalk:
“2.6 DRIVEN OFF THE TABLE
A ball is considered driven off the table if it comes to rest other than on the playing
surface but is not pocketed. A ball is also considered driven off the table if it would have been driven off the table except for striking an object such as a light fixture, piece of chalk or a player which causes it to return to the table.
A ball that contacts the top of the rail is not considered to have been driven off the table if it returns to the playing surface or enters a pocket.”
For what it’s worth, the BCA Pool League/ CSI rules have an applied ruling that makes it clear that the foul is on the shooter regardless of who owns the chalk (see 4 below) - different then their rule with regard to extensions etc as in 3 below.
“3. Situation: Player A, preparing to shoot a jump shot, unscrews the extended butt from
their jump-break cue and places it in a pocket. They then forget that the butt is in the pocket and leave it there after their inning. Player B shoots and a ball (a) rebounds from the pocket; (b) is pocketed despite the presence of the butt.
Ruling: (a) foul on Player A; (b) The result of the shot stands and the game continues.
Discussion: With the exception of chalk left on the rail, if equipment used by a
player and left at the table directly interferes with their opponent’s shot, it is a
foul on the player that left the equipment. If a player leaves equipment at the
table and it is discovered before it interferes with the game, or if it interacts with the balls during a shot without causing a disadvantage to the opponent, it
shall be removed without penalty, but a warning may be issued.
4. Situation: Player A shoots and a ball leaves the bed of the table, strikes a piece of Player B's personal chalk that they left on the rail, and falls back on the bed of the table.
Ruling: Foul.
Discussion: Personal chalk left on a rail is an exception to the responsibility
principle of Rule 1-3-1 as applied in Situation 3 above. Personal chalk is
considered a hazard for the shooter regardless of ownership.”