What this page is
The two problems are these.
- A model writes GeoJSON that is wrong, and nothing tells you. Coordinates in the wrong order, a longitude of 400, metres where degrees belong. Your code reads the file, your library says nothing, and the error shows up later as a point in the ocean.
- The geometry is too big to send. One detailed county boundary can use up a whole context window on its own, and you find out when the request fails.
The first half of the page is which mistakes your library will let through and why. The second half is what geometry costs in tokens, measured by running the o200k_base tokeniser over generated polygons of a known size, and those are numbers you can reproduce.
Check the coordinates yourself
Your library will probably not do it for you, and that is not a failing on its part. Shapely, GDAL, GeoPandas and Turf are loaders and geometry engines: they turn a document into an object they can work with. Deciding whether the document conforms to RFC 7946 is a different job that none of them took on.
Three cases matter most here, because they are the mistakes a model makes most often.
- Out-of-range coordinates. A model that has confused a projected coordinate for a geographic one writes numbers in the millions. Almost nothing objects, and RFC 7946 is the reason: section 3.1.1 sets no bound on either value, and section 4 allows an agreed alternative coordinate reference system that a file has no way to declare. A longitude of 181 is legal text. It is still almost certainly your bug.
- Swapped latitude and longitude. Nothing catches this, and nothing can. Reversing a pair produces a legal position somewhere else, and no validator can tell it from data you meant to write. It is the single most common spatial error a model makes.
- Unclosed rings. This one the specification does state, so a validator will catch it. A geometry engine may not: hand Shapely a ring whose last position is missing and it closes the ring for you, reports the result valid, and hands back geometry that is not the geometry you sent.
So check the numbers before they reach a library. Flag a longitude outside -180 to 180, a latitude outside -90 to 90, and any absolute value above 180. That is four comparisons, and it catches the whole first group above. Flag instead of refuse, because the specification does not make these an error and projected data is occasionally what someone meant.
What geometry costs in tokens
At six decimal places, which is about eleven centimetres, a position costs about 10.99 tokens. That figure is the one to plan around, and it makes the arithmetic simple: a shape is roughly eleven tokens per corner.
| Positions | Tokens | Bytes | Tokens each |
|---|---|---|---|
| 11 | 137 | 327 | 12.45 |
| 51 | 577 | 1,247 | 11.31 |
| 101 | 1,123 | 2,381 | 11.12 |
| 501 | 5,523 | 11,541 | 11.02 |
| 1,001 | 11,023 | 22,961 | 11.01 |
| 5,001 | 54,991 | 113,905 | 11 |
| 10,001 | 109,959 | 227,505 | 10.99 |
Leaving half the window free to work in, that gives roughly:
| Context window | Positions in half of it |
|---|---|
| 8,000 tokens | 363 |
| 32,000 tokens | 1,455 |
| 128,000 tokens | 5,823 |
| 200,000 tokens | 9,099 |
| 1,000,000 tokens | 45,495 |
A detailed administrative boundary runs to tens of thousands of positions, so one county can exhaust a 128,000 token window on its own. Simplify before sending, not after the request fails.
Cutting decimal places only helps at three points
Trimming decimals is the usual first move, and it mostly does nothing, because the tokeniser reads digits in groups of three. Going from six decimal places to five saves nothing at all. Going from seven to six saves one token on every number in the file.
| Decimals | The number | Tokens | How it splits |
|---|---|---|---|
| 1 | -73.9 | 4 | - 73 . 9 |
| 2 | -73.98 | 4 | - 73 . 98 |
| 3 | -73.985 | 4 | - 73 . 985 |
| 4 | -73.9854 | 5 | - 73 . 985 4 |
| 5 | -73.98542 | 5 | - 73 . 985 42 |
| 6 | -73.985428 | 5 | - 73 . 985 428 |
| 7 | -73.9854281 | 6 | - 73 . 985 428 1 |
| 8 | -73.98542812 | 6 | - 73 . 985 428 12 |
| 9 | -73.985428123 | 6 | - 73 . 985 428 123 |
So there are three useful precisions and nothing in between: three decimals, six, and nine. Three is about a hundred metres, six is about eleven centimetres, nine is about a tenth of a millimetre. Pick the band you need and round to the top of it. Anything finer is paid for and discarded.
Pretty printing is the other easy saving. Indented JSON costs 1.64 times what the compact form does for the same geometry, 9,056 tokens against 5,523. Nothing about the data changes.
When the geometry does not fit
Three approaches, in the order worth trying.
Simplify. Reduce the vertex count before the geometry reaches the model. The simplifier here reports the vertex count before and after, which converts directly into tokens using the table above. A boundary that renders identically at map scale often carries ten times the positions it needs.
Send the bounding box instead. Four numbers, under thirty tokens, and enough for most questions about where something is or whether two things overlap. Reach for the full geometry only when the answer depends on the shape. Note that a bbox crossing the antimeridian has a west value greater than its east value, which naive filtering reads as an empty box.
Keep the geometry out of the conversation. Give the model an identifier and let a tool do the spatial work. This is the approach that scales, and it removes the coordinate arithmetic that models are worst at.
Do not truncate. Cutting a GeoJSON document short gives you a syntax error in place of partial data, and the failure arrives as a JSON parse message that says nothing about geometry. Of the 61 test files in the corpus, the malformed-JSON ones produce the least useful errors of any category.
A schema will not catch these mistakes
Most model providers let you hand over a schema so the reply is guaranteed to match a given shape. That is worth doing, and it stops a different class of problem than the one above. A schema checks structure: the right members, the right types, numbers where numbers belong.
It cannot check that a polygon outline closes, or that a hole sits inside the ring meant to contain it. Both compare one value in the document against another, which is outside what a schema says. A coordinate bound is a different matter. A schema can carry one, and the schema published here does not, so a longitude of 181 goes through untouched.
The schema, what it catches, and the five rules that get past it are set out on their own page, measured against the same corpus.
Reproducing these numbers
Every token figure here was counted with tiktoken 0.12.0 and the o200k_base encoding. What was counted is a single Polygon feature holding a closed ring of evenly spaced positions, written as compact JSON with no spaces between values. Changing how many positions the ring carries, and how many decimal places each one keeps, produces the tables above. Other model families tokenise digits differently, so treat the exact figures as the right order of magnitude and not as universal. The step behaviour in the precision table is a property of how that tokeniser groups digits, and is worth rechecking against whichever one you use.
Both datasets are downloadable from the data page under CC BY 4.0.