The former trade union official had served as the Labour Party’s most senior employee since 2011, with responsibility for recruiting party staff, overseeing campaigns, organising annual conferences, and guarding Labour’s legal and constitutional structures.
Having been appointed to the role during the party leadership of Ed Miliband, Mr McNicol went on to experience a turbulent time under current party leader Jeremy Corbyn.
During an attempt by Labour MPs to remove Mr Corbyn as leader in 2016, Mr McNicol was reportedly accused of trying to stop the left-winger from being able to stand for re-election.
The row subsequently led to reports Mr Corbyn’s allies were plotting to dislodge Mr McNicol as the party’s chief, amid Labour’s turmoil in the wake of the EU referendum result.
Mr McNicol’s departure prompted suggestions Mr Corbyn’s supporters have now further tightened their grip on Labour’s internal machinery.
Last month, three candidates backed by Corbyn backing group Momentum were elected to Labour’s ruling National Executive Committee (NEC).
It has raised expectations an ally of Mr Corbyn will now be appointed as Mr McNicol’s successor.
Swansea West MP Geraint Davies, who backed Mr Corbyn’s rival Owen Smith during the 2016 Labour leadership contest, urged the party to choose a successor to Mr McNicol who “works for unity across the broad church of the party to deliver victory & success for all our communities”.
Fellow so-called Labour “moderate”, Streatham MP Chuka Umunna, said Mr McNicol had “done a smashing job for our party and our politics, and has had to put up with a lot of flak for doing so”.
He added: “We owe him a huge debt of gratitude.”
Announcing his resignation late on Friday, Mr McNicol described being Labour’s general secretary as “an absolute honour and a privilege”.
He said in a statement: “I have now decided to move on to pursue new challenges in the service of the Labour Party and wider labour movement.
“I would like to thank our fantastic staff across the country for their brilliant work and support. A team who continuously go above and beyond to deliver for Labour.”
Mr McNicol added: “I will continue to support Jeremy Corbyn’s leadership and will work closely with him and our whole parliamentary team to take Labour to victory at the next general election.”
Mr Corbyn thanked Mr McNicol for his “long and dedicated service”, adding: “He has run our party’s organisation at a time of great change, including a near tripling of the membership, two general elections and the EU referendum.
“Iain has served the Labour Party and trade union movement in a wide range of roles throughout his career and is a credit to our movement.
“I am confident Iain will continue to play a major role in politics and our party, and will support our team inside and outside Parliament with his experience and skills.”
Labour officials will meet to decide the process for the election of Mr McNicol’s successor.
A party spokesperson said: “To maintain stability, Iain will work with our NEC Officers to deliver a smooth and proper transition ensuring that the Labour Party remains on an election footing.”