100 articles · 8 breeds · tools · topics

Scripting against it

Formats and reuse

There is no API here, and that is deliberate. Everything is a static file on a CDN: no key, no rate limit, no service of ours for your script to depend on.

01 · In numbers

What is on offer

Both file sizes are built rather than typed. They were literals here until 2026-08-22, in decimal where every other page on the site reads binary, so this page and the series pages disagreed about the same file by 5 KB.

Series

59

Each with its own page and files.

Files described

237

4 per series, plus the two stacked archives.

Whole archive, Parquet

237 KB

All 59 series, typed, with real nulls. One request.

One series, CSV

66 KB

hog-farmgate-household. Most are this size or smaller.

02 · Three lines to a dataframe

Copy one of these

The real URLs, interpolated at build time. Each loads the whole archive — the stacked file is the interesting one, and filtering it on `series` is the line you would have had to work out anyway.

pandas

import pandas as pd

df = pd.read_parquet("https://baboyph.com/data/baboyph-industry.parquet")

# Typed columns, real nulls, and the unit on every row — group by both,
# never by series alone: these series are not in the same unit.
print(df.groupby(["series", "unit"])["value"].describe())

DuckDB

-- Straight off the URL. No download step, no key.
SELECT series, unit, max(date) AS newest, count(value) AS published
FROM read_parquet('https://baboyph.com/data/baboyph-industry.parquet')
GROUP BY series, unit
ORDER BY published DESC;

R

# One series' own CSV — R reads CSV natively, and NA is its default
# na.strings, so a missing figure reads as NA and never as 0.
df <- read.csv("https://baboyph.com/data/hog-farmgate-household.csv")

plot(as.Date(df$date), df$value, type = "l")

# For the whole archive at once, install arrow and read the Parquet:
# df <- arrow::read_parquet("https://baboyph.com/data/baboyph-industry.parquet")

03 · The URLs

Every one of the 59 series has the same shapes

Substitute the series id, which is the last segment of its page URL. Every row below is a live link, so you can see what one returns before you write against it.

URL What it returns
/data/hog-farmgate-household The series page. The id is its last segment.
/data/hog-farmgate-household.csv Published rows, every geography. Safe to average.
/data/full/hog-farmgate-household.csv Every row ever held, with a status column.
/data/hog-farmgate-household.json The same rows, plus the coverage summary and the licence.
/data/hog-farmgate-household.parquet The published rows, typed, with real nulls.
/data/baboyph-industry.parquet All 59 series stacked, typed, 237 KB.
/data/datapackage.json Every file above, described for a machine, with a sha256 each.
/data/croissant.json The same, as MLCommons Croissant 1.1.
/data/catalog.jsonld The same, as a W3C DCAT 3 catalogue.
/data/changelog.xml Atom feed of every retrieval that brought in a new document.

04 · Which file to take

And which not to

CSV, plain against full

The plain CSV excludes withdrawn rows, so a mean taken over it is correct without filtering. The full/ CSV keeps them and adds a status column — take that one if you are auditing this archive, not if you are computing on it. The JSON is the full set as well, because anything reading JSON can read a flag; anything reading CSV probably drops it straight into a spreadsheet and averages the column.

Parquet

Same rows as the plain CSV, with two things a CSV cannot carry. The numeric columns are doubles, so nothing has to guess. And a missing figure is a real null instead of the NA token — you can average a column without knowing what this site writes for "not published". Snappy compression, which every reader decodes without an extra package.

The CSVs arrive uncompressed

Measured off the live site rather than assumed. Cloudflare's default compressible-type list does not include text/csv, so a CSV really is as many bytes on the wire as it is on disk, while the JSON descriptors are brotli'd to a fraction of theirs. That is why the stacked CSV was withdrawn in August 2026 — 20.7 MB, uncompressed, of rows the 237 KB Parquet already carried.

There is no full-history Parquet

Deliberate. The format's whole advantage is that you can compute on it without thinking, and the file with retracted rows in it is the one you must not compute on without thinking. Audit from full/ and the status column.

05 · The three descriptors

One archive, three vocabularies

Three different kinds of tool ask three different questions. All three are generated from the registry on every build, so none of them can drift from the files.

datapackage.json — what is in the file

A Frictionless Data Package describing all 237 downloads at once: the column types, which columns can be null and what a null looks like, the byte length of each file and its sha256. Point frictionless, dpkit or pandas at it and you get typed columns without writing a parser.

croissant.json — what an ML toolchain loads

MLCommons Croissant 1.1: one record set per series with typed fields. It carries no checksums, deliberately. Croissant's own rule is not to publish a hash on a file you expect to update, and the weekly series update most Mondays — a stale hash would make every consumer throw. The hashes live in the Frictionless descriptor, which is regenerated with the files and re-checked against them on every build.

catalog.jsonld — what a harvester reads

A W3C DCAT 3 catalogue: 59 dcat:Dataset nodes, 4 distributions each, every one carrying how often the source refreshes it and the interval it covers. It lists the 59 series and nothing else — the stacked baboyph-industry files are a repackaging of the same rows, not a sixtieth series, and a harvester that ingested both would count every row in this archive twice.

On the hashes: taken from the same bytes the build wrote, and re-checked against the built files on every commit. So if a hash disagrees with what you downloaded, you have found a real difference — a CDN edge serving something stale, or a file that changed between your two requests. The sizes and hashes move whenever a row lands, which for the weekly series is most Mondays. Pin nothing to them.

06 · The columns

A published interface

This order is asserted as a literal string in this site's own test suite, so a change to it fails our build rather than yours. The descriptions below are the ones datapackage.json publishes — the same text, from the same place, so a machine and a reader cannot be told different things.

The header, verbatim

series,date,period,value,unit,geo,kind,rangeLow,rangeHigh,source_id,source_url,accessed_on,derivation,note,citation
Column Type What it is
series string The archive id of the series this row belongs to, resolvable at https://baboyph.com/data/<series>. Column one of every file so that a single-series download and the whole-archive download have the identical shape.
date date The date the reading REFERS TO, not the date it was retrieved. For an annual series this is the resolved reference date: a 31 March inventory filed under 2026 exports as 2026-03-31, not as the year end.
period string The publisher’s own label for the period, such as Q1 2026 or 2026-07. Kept verbatim so a row can be found again in the publication it came from.
value number NA means the source published no figure for this period. It is never 0 for unknown, never carried forward and never interpolated. Its unit is in the unit column and differs across series, so grouping by series without reading unit gives a nonsense answer.
unit string What the value is measured in, as the source declares it: PHP/kg liveweight, head, metric ton. Carried on the row rather than in the filename because units differ within this archive and, for two PSA production tables of the same commodity, between tables.
geo string The geography the reading covers. Empty on a series that has only ever had one. Codes and names follow PSA’s livestock vocabulary; https://baboyph.com/data/geography resolves them, and PSA’s retail price tables use a different numbering that is not translatable by arithmetic.
kind string How the figure was arrived at: which survey, or that it is derived. Empty on the oldest hand-built rows.
rangeLow number Set when the source published a range. With a value present, the value is the midpoint and this pair is what stops that midpoint being read as a measurement; with value NA, the source published a range and this archive publishes no single figure for it.
rangeHigh number See rangeLow. The two are always set together or not at all.
source_id string The publisher this row was read from, resolvable in the archive’s citation registry and rendered at https://baboyph.com/data/provenance.
source_url string The document or query this row came from. Swept monthly for liveness; a URL that has rotted is recorded and never causes a row to be withdrawn.
accessed_on date The most recent date the source was read for this row. Where it differs from the first reading of that row, the row has been re-read and still says the same thing.
derivation string Empty on a retrieved row. On a derived one it states what was computed from what: a synthesis figure without its arithmetic is exactly the unsourced number this archive exists to prevent.
note string Anything about this row a reader needs before using it. Prose, and the one place in a row where non-ASCII appears.
citation string The attribution line, repeated on every row. The unit of loss is the row, not the file: somebody pastes four lines into a group chat and a file-level header does not survive that.

07 · Four things that will bite you

Read these before you write the loop

1

A missing value is the token NA, not an empty field.

An empty field is 0 to Number(''), which turns "PSA has not published July yet" into "₱0.00/kg" with no error anywhere. NA is R's default na.strings, is in pandas' default na_values, and is NaN in JavaScript. Loud in all three.

2

Read the unit column.

These series are not in the same unit, and the difference between per-kilo and per-sack is a factor of fifty. Grouping by series without it gives a nonsense answer that looks fine.

3

Do not average two sources.

PSA and BAI both survey hog farmgate prices and they disagree. Both are here as separate series, on purpose; a mean of them is a price no survey measured.

4

Line endings are CRLF and there is no byte-order mark.

Text fields are always quoted, whether or not they contain a comma, so the quoting does not change when the data does. A field that would start with = or + — which a spreadsheet executes — fails our build rather than being silently rewritten, so you will never receive one.

08 · Licence

Attribution is the only condition

CC BY 4.0. Use it, republish it, build a product on it. The figures are official statistics from PSA and BAI, which are public; what this archive adds is the retrieval, the provenance on every row and the refusals. The attribution line is on every row of every file for that reason:

The line on every row

Baboy PH state-of-the-industry archive, series "hog-farmgate-household". CC BY 4.0. https://baboyph.com/data/hog-farmgate-household

Note that it is a COLUMN, not a header. Every row carries its own, because the unit of loss is the row: a file-level notice does not survive somebody pasting four lines into a spreadsheet. Leave the column in place and the attribution travels with the data wherever it goes, including into work that forgot where it came from.

Everything outside /data — the articles, the breed guides and the calculator assumptions — is not under this licence. Quote it with a credit and a link; the terms say so in one short section.

09 · Every series id

All 59 of them

What the archive does NOT know is documented too — the gaps page lists the numbers nobody publishes and what we checked before saying so.