No! The listing agent remarks are viewable to agents when the agent is logged in. The agent will see the private remarks on the front end home buyer search, if they never logged out as an agent.
You do not have to worry! Your home buyers NEVER see the the listing agent remarks or contact information. Below are photos depicting examples.
Home Buyer Search View:
Agent Search View: