What sets the limit
One minute of arc in a sextant altitude is one nautical mile on the chart. So the question is really how well you can measure an angle from a moving deck, and how well you know the time.
- The sight itself. On a small boat the horizon is a line of wave tops and the deck is moving. A minute or two of scatter between sights is normal. This is the largest error by far.
- Time. Four seconds of watch error moves a line by up to one mile (one minute of longitude at the equator, less toward the poles).
- Index error not checked, or applied with the wrong sign: a constant error in every sight.
- Height of eye and dip. A wrong height of eye changes dip a little; unusual air temperatures near the sea can change it by a minute or more.
- Refraction at low altitudes. Below about 15° the correction is large and uncertain. Avoid low sights when you can.
- Tables and plotting. Pub. 249 rounds to a minute; a pencil line has a width. Usually under a mile together.
- Dead reckoning, in a running fix: the run between two Sun sights carries any error in your course and speed.
How navigators shrink the errors
- Take several sights of each body and average them, or plot them against time and discard the one that does not fit.
- Shoot three stars about 120° apart. A constant error, such as index error or abnormal dip, then moves each line the same way, and the true position stays at the center of the triangle.
- Shoot at twilight, when the horizon is sharp and stars are bright enough: the first half of evening twilight, the second half of morning.
- Check the index error at every round of sights, and the watch against a time signal daily.
- Keep altitudes between about 15° and 70°.
How it compares
GPS is good to a few meters; celestial to a few miles. From a ship's steady bridge with a good sextant, practiced navigators regularly get within a mile. The value of celestial navigation today is that it needs no power, no signal and no satellites: with a sextant, a watch and the books, you can always find where you are.
See it for yourself
In the course, the lessons on the fix show how the cocked hat grows and shrinks with the spread of bearings, and the practice passages let you sail a day at sea and see how close your fixes come. With your own sextant, work your own sight and compare it with a known position.