- Python
- REST API (CEPiK)
- PostgreSQL
- resumable ETL
Counting every Corolla in Poland
Poland's national vehicle registry will tell you, exactly, that 89 450 cars of one platform are on the road: 24 992 Corollas and 64 458 of its twin, the Auris.
Poland publishes its vehicle registry as open data, around 18 million vehicles. I needed one model family out of it. Scanning everything would have taken most of a day, so I wrote a probe script first and proved the API supports a server side make filter. That cut the national census to about half an hour and, more importantly, made the count exact rather than sampled.
The collector is resumable, with a checkpoint per window, and a window only commits once it has been fully paged. Government APIs drop connections, and a census that cannot be resumed is a census you will never finish. Aggregates only: the raw rows never leave the machine, there is no personal data, and every surface that shows a number credits the source.
KEY SPEEDUP: CEPiK /pojazdy supports a SERVER-SIDE make filter `filter[marka]=TOYOTA` (proven by probe_cepik_filter.py). So instead of scanning all ~18M vehicles to find Toyotas, we pull ONLY Toyotas (~7-13% of the fleet) -> the full national census runs in ~20-30 min instead of ~6-10 h, and the count is EXACT (not sampled). CTX = ssl.create_default_context(); CTX.set_ciphers("DEFAULT@SECLEVEL=1") # CEPiK weak DH key PROD_YEARS = {str(y) for y in range(2013, 2020)} # E170 generation 2013-2019 LIMIT = 500 SLEEP = 0.7 # < 100 req/min, < 20 req/s
What the census actually found
Every Toyota on Polish roads, by model, all generations. The two highlighted bars are where the platform I wanted was hiding.
Source: CEPiK national census, full pass, 1 070 191 Toyota records. Bars are all generations; the E170 platform slice inside them is 24 992 Corolla plus 64 458 Auris.