Mr Bellingham, MP for North West Norfolk, said he was gutted to be asked to leave by Speaker Michael Martin after being reported by a clerk for using his mobile in the House.
"I think it was a bit harsh to be given a red card straight away, normally you would get a yellow card first," said Mr Bellingham, who had to leave the Chamber immediately.
He claimed he had not been using the phone, which has a built-in camera, but had simply been "fiddling" with it to check his messages.
"I know you are not supposed to have a camera or use your mobile in the House, but most MPs check their text messages, e-mails or caller logs on their phones, or have their bleepers on," said Mr Bellingham.
"A clerk saw me with my phone and realised it had a camera and reported me to the Speaker. I did not realise there was a new clampdown on phones in the House and I feel a bit gutted that I am the first victim of this new policy.
"However, I do feel suitably chastened and I have written a letter to the Speaker apologising," said Mr Bellingham.
His expulsion came on the last day of Parliament before the Christmas recess and he said: "It is sad to be in trouble on the last day of term, but you don't argue with the Speaker – the same as a footballer doesn't argue with a referee when he gives a red card. You just have to accept it."
Mr Bellingham was barred from returning to the House for the rest of the day.
"It didn't really matter as I had already asked my question and it wasn't like I was missing an important debate – it's just disappointing to get into trouble in that way," he said.