A car accident in December ended his participation this season.
Royce told the Bees' official website: "This is a great opportunity for me as I start to wind down my playing career and start a new chapter in my life."
He added: "I'm delighted to be working with Andy Scott, who I consider one of the brightest young managers in the league."
Brentford boss Scott said: "Simon fills a role that will grow ever more important as we strive to progress the club.
We might bring in a part-time goalkeeping coach depending on the budget
Gillingham manager Andy Hessenthaler
"His ambition for success and desire to be a respected coach were obvious from our very first conversation."
Gills manager Andy Hessenthaler told BBC Radio Kent: "It's unfortunate that we're going to lose Simon. But he's got a better opportunity in a division above and wants to move on."
"We offered him a role at the club as goalkeeping coach, plus we wanted him to push Alan Julian - who's still number one at the moment.
"We're disappointed with that because he's experienced - he would have been great as well for Alan Julian on the coaching side of it so we'll now be looking for another keeper.
"That's another position we're going to be working on in the next few weeks.
I am very disappointed at Roycie, I thought he was Gills through and through
gillsms
"We might bring in a part-time goalkeeping coach depending on the budget - we're looking at all different things so we won't necessarily bring an older keeper in."
Royce first joined the Gills on loan at the end of the 2006/07 season and was a regular in the side that won promotion to League One in 2008/09.
Meanwhile, Gillingham winger Dennis Oli has followed captain Barry Fuller in signing a new two-year contract.
Source: BBC Sport
Source: BBC Sport