Rub them to change their destination, then invoke them to go there. The destinations are color-coded, but it's easier to just jump through all the locations until you get where you're going. The fruit stand is no-teleport, so you'll need to walk out of that room before invoking.