The Roche limit (or Roche radius) is the nearest to an astronomical body that a body orbiting it can approach before breaking up. A moon within its planet's Roche limit is being pulled apart by the planet's gravity more than its own gravity is holding it together. Thus at this limit, moons (or in the case of stars, its planets) tend to break up into rings (like Saturn's) or circumstellar disks. The exact formula depends upon a few factors, but a basic formula is:
d = Rm(2 MM/Mm)1/3
This formula is derived by determining the distance at which the gravitational force and tidal force balance. At longer distances, the secondary body's gravity dominates, holding it together, and at shorter distances, the tidal force from the larger body overcomes its gravity and pulls it apart.
Roche lobe means roughly the same thing as Roche limit, but likely refers to the region within the limit, which has a teardrop shape. Roche lobe overflow (RLOF) refers to a body exceeds its Roche lobe and material is pulled off, such as an interacting binary star.