Conservative peer Lord Hamilton has apologised after saying the Jewish community should “pay for their own” Holocaust memorial because they have “an awful lot of money”.

The former minister said his comments in a House of Lords debate on plans for a memorial near Parliament were “insensitive” but “not intended to be antisemitic”.

A Conservative Party spokesman condemned the comments as “completely unacceptable” and “flagrantly antisemitic”.

Responding to calls for him to be suspended from the parliamentary party, leader Kemi Badenoch said: “I have intervened in the situation. Lord Hamilton misspoke and he’s apologised and I think that should be the end of the matter.”

Speaking during a debate on legislation to allow a Holocaust memorial and learning centre to be built in Westminster, with public funds used to pay for it, Lord Hamilton said: “I do not understand why the government have volunteered taxpayers’ money, when there is so little of it, to finance this.

“The Jewish community in Britain has an awful lot of money. It has a lot of education charities that would contribute towards this. I do not understand why they should not pay for their own memorial.”

Challenged by Lord Austin that the memorial was “for everybody” not just the Jewish community, Lord Hamilton added: “I take that point, but the driving forces behind putting up this memorial are the Jewish people in this country.

“They are people who have property everywhere. I do not see why they should not fund it.”

The former minister under Margaret Thatcher and John Major withdrew his comments the following day and said in a statement: “My remarks were not intended to be antisemitic and I apologise unreservedly.

“With retrospect having read my comments, I realise that my remarks were insensitive and I apologise. I intend to meet with groups from the Jewish community in order to better understand how to communicate about issues affecting British Jews.”

His original remarks were criticised by the Antisemitism Policy Trust, who said they were “ill-judged, racist and false”.

During the debate, Conservative peer Lord Finkelstein said: “The Holocaust is not something that is just about the Jews.”

He added that Lord Hamilton’s comments were “probably a pretty eloquent contribution as to why we need this memorial – and near Parliament”.

Former Labour MP and crossbench peer Lord Austin described the remarks as “disgusting racism” and called on the Conservative Party to take “proper action”.

“The Tories were quick to demand Labour took action on antisemitism and quite right too,” he wrote on social media.

“But they must now do the same themselves.”

Plans for a Holocaust memorial in Victoria Tower Gardens in Westminster were first announced under Conservative Prime Minister David Cameron in 2016.

A bill to remove restrictions on the memorial being built in the gardens and to allow public funds to be used for it is currently making its way through Parliament.