Isolation mass is the mass of a planet-to-be created by accretion of planetesimals (objects with a diameter on the order of 1km) by the largest planetesimals (oligarches). Current models set it at roughly lunar mass.
Formation of isolation mass objects is a step in the nebular hypothesis of planet formation, where planetesimals (1km diameter) form by some means, the largest (oligarchs) accrete the rest, then the oligarchs' orbits interfere and become unstable, the oligarchs collide and merge into rocky planets such as Mars or Earth. The model was developed before the identification of extra-solar planets, and since-discovered planetary systems add challenges to the concept. An isolation mass planet is a planet that adheres to the concept, and an isolation mass core is a planetesimal that has grown to the isolation mass, the name suggesting the possibly that it might grow further through impact or gas accretion.
The isolation mass is determined by the size to which the Hill radius can grow as the object grows, naturally, the density of material falling within it being a factor. Given that in addition to the gravitational force that determines the Hill radius, there also exists gas and radiation pressure, a stability radius that takes these into account offers a more precise model.