Zastron Labs

All work

01

Counting every Corolla in Poland

Poland's national vehicle registry will tell you, exactly, that 89 450 cars of one platform are on the road: 24 992 Corollas and 64 458 of its twin, the Auris.

  • Python
  • REST API (CEPiK)
  • PostgreSQL
  • resumable ETL

Poland publishes its vehicle registry as open data, around 18 million vehicles. I needed one model family out of it. Scanning everything would have taken most of a day, so I wrote a probe script first and proved the API supports a server side make filter. That cut the national census to about half an hour and, more importantly, made the count exact rather than sampled.

The collector is resumable, with a checkpoint per window, and a window only commits once it has been fully paged. Government APIs drop connections, and a census that cannot be resumed is a census you will never finish. Aggregates only: the raw rows never leave the machine, there is no personal data, and every surface that shows a number credits the source.

backend/data/collect_cepik.py
KEY SPEEDUP: CEPiK /pojazdy supports a SERVER-SIDE make filter `filter[marka]=TOYOTA`
(proven by probe_cepik_filter.py). So instead of scanning all ~18M vehicles to find
Toyotas, we pull ONLY Toyotas (~7-13% of the fleet) -> the full national census runs
in ~20-30 min instead of ~6-10 h, and the count is EXACT (not sampled).

CTX = ssl.create_default_context(); CTX.set_ciphers("DEFAULT@SECLEVEL=1")  # CEPiK weak DH key

PROD_YEARS = {str(y) for y in range(2013, 2020)}     # E170 generation 2013-2019
LIMIT = 500
SLEEP = 0.7                                  # < 100 req/min, < 20 req/s
1 070 191Toyota records scanned
89 450cars on the E170 platform
16regions covered
0raw rows leaving the machine

What the census actually found

Every Toyota on Polish roads, by model, all generations. The two highlighted bars are where the platform I wanted was hiding.

Yaris Corolla RAV4 Auris C-HR Avensis 260 037 225 468 111 403 89 558 87 041 70 051

Source: CEPiK national census, full pass, 1 070 191 Toyota records. Bars are all generations; the E170 platform slice inside them is 24 992 Corolla plus 64 458 Auris.

How it was done

  1. Prove the shortcut before building the long way round

    A separate probe script established that the registry API accepts a server side make filter. Without that, the job means paging the whole national fleet. With it, the job is only Toyotas. The probe is a few minutes of work that decides whether the pipeline takes half an hour or most of a day.

  2. Make the collector survivable

    The scan is windowed, and each window writes a checkpoint. A window is only committed once it has been fully paged, so an interruption is redone cleanly rather than leaving a half counted window that quietly corrupts the total.

  3. Stay polite to a public service

    Requests are paced deliberately below the published rate ceiling, and the client is identified. A census that gets the source blocked is not a census.

  4. Aggregate at the edge, then throw the rest away

    The detailed rows are processed into counts by year, region and fuel, and only those aggregates are stored. The raw pull never enters the repository and never leaves the machine.

What this does not tell you

Every dataset has an edge. Here is where this one stops, said plainly, because a number you have to qualify is worth more than one you do not.

  • It counts vehicles currently on the register, which is not the same as vehicles on the road. Cars that are off the road but still registered are included.
  • The 2019 build year is deliberately excluded from the Corolla figure, because the next generation went on sale that year and counting it would inflate the platform. That single decision is the difference between the honest 24 992 and a flattering 45 397.
  • The model mix chart shows all generations of each model, not just the platform I was after. It is there to show the haystack, not the needle.

See it on the live dashboard

Back to all six projects