Any suspicion is a legitimate reason for Border Patrol agents to detain you and that suspicion can extend to any action or inaction by you that is not sufficient to allay their suspicion. The agent just might have a hair up his ass that day and detain for that reason alone.
Also if he does not believe you are a legal citizen (despite your skin color, etc) then you are not entitled to legal defense or other customary rights accorded to US citizens. You can be detained indefinitely withough outside contact.
Border Patrol Harass Military Field Grade Officer At Internal Checkpoint
Check out the constitution free zone map:
Constitution Free Zone - Map
Point of correction: Constitutional rights, including the right to legal defense, are guaranteed to any PERSON on our soil, not just to US citizens.