Rory Stewart has said the move would ease pressure on prisons and be better for offenders.
In comments reported by The Daily Telegraph, he said that “very short” jail terms were “long enough to damage you and not long enough to heal you”.
He added: “You bring somebody in for three or four weeks, they lose their house, their job, their family, their reputation.
“They come [into prison], they meet a lot of interesting characters (to put it politely) and then you whap them on to the streets again.
“The public are safer if we have a good community sentence… and it will relieve a lot of pressure on prisons.”
Around 30,000 offenders a year including some burglars and most shoplifters could be spared jail under the proposals.
Mr Stewart said that the exception to his plans would be violent offenders or those convicted of sex offences.
He told the newspaper that such a move would free up 4,000 prison places and create more space for education and workshops, adding: “We can change lives more”.
The number of prisoners has doubled since the 1990s to around 80,000 and around 60% of short-term prisoners re-offend within a year of their release, the Telegraph says.
The Prison Reform Trust says that almost half of prisoners in England and Wales have been sentenced to six months or less.
In a report last year called Prison: The Facts, the trust said short prison sentences were less effective than community-based sentences, yet use of the latter had more than halved in the previous decade.
Peter Dawson, director of the Prison Reform Trust, said: Ministers should be congratulated for having the political courage to start the debate.”
Steve Gillan, general secretary of the Prison Officers Association, said: “I think Rory Stewart is flying a kite to see how it goes down – I don’t think the public or his party will go for it – and I don’t think it will happen.
“The real problem is about resources, and this government has been starving the police, prisons and probation services of resources for years.”
Ian Lawrence, from the probation officer’s union Napo, has broadly welcomed the plans.
But he added that probation services in the UK are in a “state”, and that the government needs to offer more resources if officers are going to be taking on more offenders.
A Ministry of Justice spokeswoman said: “As we have said previously, short sentences are too often ineffective, provide little opportunity to rehabilitate offenders and lead to unacceptably high rates of re-offending.
“That’s why we are exploring potential alternatives but this work is ongoing and we have reached no conclusions at this time.”