Fleets attempt to escape according to the settings specified on the fleet command page (namely, the fleet's Stance) , your fleet may choose to run if the odds are overwhelming.

The chances of escaping are based off of the fleet's speed and the fleet speed of the opposing fleet. The faster you are than your opponents fleet, the more likely you are to escape.

If you have two or more fleets in battle, it is much easier to pull one out than if you only had one fleet in the battle. You can send in a second fleet to act as a decoy so that you can pull out the other fleet out. This can be useful if you want to save a fleet that faced certain destruction.

Note that a fleet can automatically run away from a battle at a colony, since the colony certainly isn't going to come after you. However, if the colony has any ships in it, your fleet will not have a 100% chance of running away.