A demolition contractor says residents' fears about asbestos dust from work at the Mike's Grill and Bar site on Princes Highway, Sylvania, are unfounded.
The restaurant, which has been derelict for three years after a fire in August, 2016, is being redeveloped as a gym.
Roy Mardian, the owner of Demo Connect, said an asbestos hygienist report received prior to work starting revealed "only a few little speckles in the kitchen, which we carefully removed."
Mr Mardian said workers put up a sign, reading "Danger Asbestos Removal in Progress" to keep people off the site.
"[Workers] wore asbestos clothing because one of them was worried about dust," he said.
A resident in Koorooma Place, behind the work site, told the Leader residents were not notified asbestos was to be removed before work started on Monday.
"The workers wore protective clothing and masks and began throwing material onto the open ground in the rear car park of the building," he said.
"The material was not covered.
"Later, a truck and mini front end loader arrived and began lifting and loading the material.
"The building is clad in what appears to be asbestos sheeting, and the majority of this material has been placed on black plastic sheeting."
Mr Mardian said his workers were stripping out the building so it could be turned into a gym.
"No walls are being taken down," he said.
"Most of the material we are removing is Hardiflex."
Mr Mardian said, as required, an online notification was made to SafeWork NSW before work began.
"We do everything by the book. Our licence is on the line, so we are not going to do anything to put that in jeopardy," he said.