The 4 is history and the foul is on the 5 according to APA and BCA rules.
I lost an MVP match in APA because of that exact thing. Opponent missed, I got up and shot the four, then the five, then the six, THEN my opponent called it on me. Pretty sure he knew it from the beginning.
APA rules state the foul must be called PRIOR to next shot, getting down a ball wouldn't do it, I read the rules as you must stroke the CB to complete the next shot. So, if I would have noticed and SHOT at the 3 then he couldn't have done anything about it.
Sort of a similar situation in tournament using BCA rules. Opponent missed and left me with the 8B (this is an 8B match) sitting IN the corner pocket and CB about 1.5' away, thought I heard him say "that's good", it was winner rack so I racked the balls and broke. Opponent THEN said he didn't give me the game. Since it was game one of a race to two we just started over. BCA rules state the foul must be called PRIOR to the next shot, just like APA but I was new and didn't know that. Had I known then what I know now I would have told him to pound sand. Sure, he didn't "give" me the game, so he says, but he didn't stop me from racking and breaking either. Oh, to top it off, 10 man tourney, $10 entry, $100 purse and they only paid to 2, I place 3rd and as mentioned I am new and have been kicking everyone's ass in league so I'm pretty sure they said F him, we're only paying to 2. I've never been in a tourney of that size that didn't pay 60/30/10.
Haven't played in that tournament since.