To avoid these types of issues, league directors typically adapt specific rulings/definitions. This is common enough regardless of league so it should be defined. A league (and/or tournament) director can interpret a specific ruling as they choose for their league/tournament.
I understand the dilemma as Saratoga is a call ball/call pocket game; however, in the absence of a prior ruling, the same application of rules should apply as with all open tables regardless of the intended object ball. Additionally, in the absence of a referee, the cue ball should not have been picked up without an acknowledgement of a foul.
The rules state "You have to contact the lowest ball of your kind (lowest of either one if table is open)". In your case, it is not a foul; however, if the one ball was pocketed it would still be an open table and would fall under the legal slop rule...lost of turn and the incoming player has the option of giving the shot back.
Same as BCA while shooting at a stripe on an open table; but contacting a solid first instead. A pocketed ball doesn't count as it wasn't the called intent; however, it isn't a foul as long as the rail/contact rule was met on a "open table". You simply can't hit your opponents suit if the table is open and suits haven't been declared but you still have to meet the call ball/pocket rule.
Shorter version:
No foul on the shooter.
The breaker fouled once he picked up the cue ball.
Hopefully, they simply enjoyed the debate and agreed to re-break the rack.