The Roche limit (or Roche radius) is the nearest to an astronomical body that a particular body orbiting it can approach before breaking up due to tidal forces. A moon orbiting a planet within their Roche limit is being pulled apart by the planet's gravity more than its own gravity is holding it together. Thus at this limit, moons (or in the case of stars, its planets) tend to break up into rings (like Saturn's) or circumstellar disks. The exact formula depends upon a few factors (including the masses of both objects), but a basic formula is:
d = Rm(2 MM/Mm)1/3
This formula is derived by determining the distance at which the gravitational force and tidal force balance. At longer distances, the secondary body's gravity dominates, holding it together, and at shorter distances, the tidal force from the larger body overcomes its gravity and pulls it apart.
The term Roche sphere is generally used as a synonym for Hill sphere, which applies to the influence of the host's gravity as if tidal forces were not a factor: the Roche/Hill sphere's radius is not this Roche radius, but is termed the Hill radius.
The term Roche lobe is more related to the Hill radius than the Roche limit, (the Roche lobe is not defined by tidal force): it indicates the actual elongated (non-spherical) regions of orbiting body's gravitational influence that meet at their L1 Lagrangian point.