Write a dataframe transformation
Writes pandas or polars code for a described transformation with built-in checks on row counts, nulls, key uniqueness and join cardinality. Use when reshaping, joining or aggregating data.
Dataframe code usually fails silently, not loudly: a join on a key that is not unique multiplies rows, a left join leaves nulls that later vanish in an aggregation, a string key with trailing spaces matches nothing, and dates parsed in the wrong format shift by months. The output looks plausible and is wrong. Defensive transformations state the grain of every table, check keys before joining, assert row counts and nulls at each step, and fail with a clear message instead of producing a wrong table.
Write code that turns this input:
into this output:
- State the grain (what one row represents) and the key of each input and of the output.
- Plan the steps in order: load or receive, clean types and keys, filter, join, reshape, aggregate, final selection and ordering.
- Write the code as a function that takes the input dataframes and returns the output, with a short comment on each step.
- After each step that can change row counts or introduce nulls, add a check:
- keys: uniqueness on the side that should be unique, before every join;
- joins: the expected cardinality (pandas
merge(..., validate="many_to_one"), polarsjoin(..., validate="m:1")) and a count of unmatched keys; - row counts: expected equal, smaller or larger than before, and by how much;
- nulls: in key columns and in columns the output requires;
- aggregates: totals that should be preserved (for example the sum of amounts before and after reshaping).
- Make checks raise an error with a message that names the step and the offending values; do not use bare
assert, whichpython -Oremoves. - Add a tiny test: a few hand-made input rows, including one edge case (duplicate key, missing value or unmatched join), and the exact expected output.
- Use idiomatic, vectorised : for pandas, method chaining where it stays readable,
.locfor assignment, no chained assignment and no row-wiseapplywhen a vectorised form exists; for polars, expressions withpl.col, and the lazy API for large data. - Write code compatible with current stable releases, and name any feature that needs a recent version.
- Do not guess column names, types or business rules. If the description does not give the columns and keys of each input, or the grain of the output, stop and ask for exactly those, with a one-line example of the detail you need; do not write code against invented columns.
- For smaller gaps (for example which duplicate to keep, or how to treat unmatched rows), choose the safest behaviour, list it under Assumptions, and make it easy to change.
- Keep it self-contained: imports at the top, no reading from paths you invented; take dataframes as parameters.
Assumptions
Bullets: grains, keys and every assumption made.
Code
One fenced Python block with the function and its checks.
What the checks catch
A table: check | step | the failure it prevents.
Test
A fenced Python block with the small test and its expected output.
2 required values still a placeholder; the assistant will ask for them.
details
- kind
- Prompt: a task you run by name to get one finished thing back
- domain
- Data analysis
- category
- Data exploration
- level
- Intermediate
- made for
- Data analyst, Data scientist, Data engineer
- risk
- read-only
- version
- v1.1.0 · experimental
- reviewed
- 2026-10-02
- works in
- Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Antigravity, OpenCode, Windsurf, Zed, Continue, AGENTS.md, ChatGPT, claude.ai
use in
npx @hermes-hq/hodios install write-dataframe-transformation --target claude-codenpx skills add hermes-hq/hodios-dist --skill write-dataframe-transformation -a claude-codeclaude plugin marketplace add hermes-hq/hodios-distclaude plugin install hodios-data-analysis@hodiosThe plugin brings every entry in this domain at once.
pairs well with
All of Data explorationReconcile two datasets
Reconciles two datasets that should agree, such as bank versus ledger or CRM versus billing, by matching records, listing mismatches and explaining likely causes. Use for month-end checks.
reconcile-datasetsConsulting statistician
Consulting statistician who asks how the data were produced before analysing them, chooses methods that fit the question, checks assumptions and refuses to over-claim. Use for any data analysis.
statisticianAnalyse an employee engagement survey
Analyses an employee engagement survey with group scores under minimum-group-size privacy rules, eNPS, comment themes and three priorities to act on. Use after an engagement or pulse survey closes.
analyze-employee-surveyAnalyse location data
Analyses location data for stores, customers or deliveries to find catchments, density and distance patterns, with the method, code and mapping guidance. Use for site, coverage or delivery questions.
analyze-location-dataCompare marketing attribution models
Compares last-click, first-click, linear, position-based and data-driven attribution on supplied channel data and explains what each implies for budget. Use before moving marketing spend.
analyze-marketing-attributionAnalyse sales performance
Analyses sales data by product, customer, region and time to find what drives revenue, seasonality, best and worst performers, and the actions worth taking. Use for a sales performance review.
analyze-sales-data