Conservatives Will Quince, Mims Davies and Anne Milton have left the Commons Reference Group on Representation and Inclusion, which is chaired by Mr Bercow.
It follows a damning report which said a culture at Westminster had allowed the bullying and sexual harassment of staff to flourish.
Dame Laura Cox QC, a former High Court judge, raised concerns over whether the current Commons leadership could get a grip on the issue.
This sparked calls for Mr Bercow, who became Speaker in 2009, to quit.
The Speaker has himself faced claims of bullying staff, allegations he strongly denies.
Mr Quince said he was leaving because he did not think the Speaker was the right person to tackle the “numerous and serious issues” raised about the conduct of some MPs and staff in the Commons.
The Colchester MP said it was a decision he had taken “with regret”, as he believed the group had made progress towards creating a more inclusive environment in parliament.
Mr Quince said: “However, in the light of the Dame Laura Cox report, sadly I cannot in good conscience remain as a member of the group while John Bercow is chair.
“As much as I personally like John, I have reluctantly reached the conclusion that he is not the right person to resolve the numerous and serious issues.”
Ms Davies echoed her Tory colleague, saying: “I felt that remaining on this (committee) currently did not sit right following Dame Laura Cox’s report, and as this committee is chaired by the Speaker, I sadly felt in a difficult position.
“I did not take this decision lightly.”
A spokeswoman for Mr Bercow said he would consider the group’s future after the House of Commons Commission has met on Wednesday to consider the report.
She said: “Dame Laura Cox’s report has highlighted some of the most significant challenges women face in our parliamentary culture.
“In the spirit of an independent approach, the Speaker feels it is right to reflect on the best means of tackling these cultural issues via the House’s response to the Cox report.
“He will therefore consider the future of the reference group following the commission meeting tomorrow.”
Labour has accepted the recommendations of the report ahead of Wednesday’s meeting.
A party spokeswoman said: “The report highlights a culture and practices that are unacceptable in any workplace.
“It is vital that this is put right as a matter of urgency.”
The party wants there to be an independent process for complaints against MPs which does not involve other MPs ruling on them, as well as the new Independent Complaints and Grievance scheme to be able to consider historical allegations.