Çatalhöyük was a very large Neolithic and Chalcolithic proto-city settlement in southern Anatolia, which existed from approximately 7100 BC to 5700 BC, and flourished around 7000 BC.
The deceased in this village where bonded closely together and placed in a hole under the sleeping area. Dr. Amy Bogaard explains this was so the living could live with their ancestors.
As a part of ritual life, the people of Çatalhöyük buried their dead within the village. Human remains have been found in pits beneath the floors and, especially, beneath hearths, the platforms within the main rooms, and under beds. Bodies were tightly flexed before burial and were often placed in baskets or wound and wrapped in reed mats. Disarticulated bones in some graves suggest that bodies may have been exposed in the open air for a time before the bones were gathered and buried. In some cases, graves were disturbed, and the individual's head removed from the skeleton. These heads may have been used in rituals, as some were found in other areas of the community. In a woman's grave spinning whorls were recovered and in a man's grave, stone axes. Some skulls were plastered and painted with ochre to recreate faces, a custom more characteristic of Neolithic sites in Syria and at Neolithic Jericho than at sites closer by.
Wikipedia page on Çatalhöyük
Çatalhöyük has also been covered in "The Story of God with Morgan Freeman" series SE01 EP03 where Dr. Amy Bogaard explains (around 9:40) that the graves would be opened for ritual purposes and to add new bodies. Youtube link 2:30
My question is what happened when the burial holes were full and no more room was in the house?
The question here may be based on some assumptions, relating the culture of the distant past to that of today. the simple-seeming question
What happened when the burial holes where full and no more room was in the house?
Is probably answered by a simple-seeming answer: They buried them somewhere else.
The issue is that we assume the burials might be related to that of those we are associated with, those of our parents or family members. Research at Çatalhöyük showed discrepancy with what we expect to be normal. From an article at LiveScience, No Family Plots, Just Communal Burials In Ancient Settlement (emphasis mine)
They found that the people buried beneath the floor of each house were, in general, not related to each other. With the possible exception of one building, this occurred throughout the entire site for as long as the settlement existed.
So the concept of 'burying ancestors under the floor' is not well understood at all, but it was probably had a much different meaning for those residents of this city 10,000 years ago than for us today.
(Though the following lectures are lengthy, they give an overview of the scale of scientific inquiry going on at Çatalhöyük, how much we are learning and how much we still don't know)
Some Ian Hodder lectures to enjoy:
The leopard changes its spots: recent work on societal change at Çatalhöyük - Prof Ian Hodder-The British Institute
Ian Hodder | What we learned from 25 Years of Research at Catalhoyuk - the Oriental Institute
Another video showed up on my recommended list today, which is very specific and reveals quite a bit concerning the problems associated with drawing conclusions from remains found. The presentation is by two members of the human remains team at Çatalhöyük , Christopher Knüsel and Eline Schotsmans. The video is House Societies, Ancestors, and Burials at Neolithic Çatalhöyük, and discusses the layers of burials, terminology difficulties with describing remains conditions, and problems trying to draw conclusions by comparing what is observed to observations from more recent behaviors.
Two sections may be of interest here, at 25:46 there is a discussion concerning an individual that was the 'last' to be buried under a specific platform.(No light is shed on what happened next, however, since we have no way of knowing.)
A later segment discusses difficulties in understanding a definition for how these individuals might have been considered 'ancestors', with a slide showing various interpretations of what that might have meant.