Barrett, who made 289 appearances in six seasons with the Shrimpers before leaving in 2010, has now been made available for transfer by Gillingham.
But Brown will not be looking to bring him back to Blues, despite his current lack of defenders.
“I’ve been made aware of Adam’s availability but he isn’t someone we will be trying to bring in,” said Brown.
“I know that he’s a legend at this football club and he’s had a brilliant career in the game.
“But the club has been down this road before, bringing back people like Freddy Eastwood, and it doesn’t always work out the way you want.
“Adam’s also 34 now and I would ideally like to bring in a younger, dynamic type of defender.”