A Python library to flatten a dictionary with nested dictionnaries and lists
Project description
dictflat
A Python library to flatten a dictionary with nested dictionnaries and lists
Use cases
Transform a dictionary structure into a new organization ready to be inserted into a relational database.
Installation
poetry add dictflat
Quick start
>>> from dictflat import DictFlat
>>> import json
>>> r = DictFlat(
root_key="root"
).flat(
d={
"name": "John",
"pers_id": 12,
"birth": {
"address": {
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
},
"date": "10/06/1976 01:10:35"
},
"Phone_Numbers": [
{"type": "home", "number": "555-1234"},
{"type": "work", "number": "555-5678"},
],
}
)
>>> print("%s" % json.dumps(r, indent=2))
{
"root": [
{
"__id": "662783f7-b1a0-4e8c-9de9-f3a72a896d4c",
"name": "John",
"pers_id": 12
}
],
"root.birth": [
{
"__id": "e72d549a-89f5-4208-99c0-4ce3493cbf9e",
"__ref__root": "662783f7-b1a0-4e8c-9de9-f3a72a896d4c",
"date": "10/06/1976 01:10:35"
}
],
"root.birth.address": [
{
"__id": "cc489c03-82ca-4b6e-a620-32c9c4be236c",
"__ref__root.birth": "e72d549a-89f5-4208-99c0-4ce3493cbf9e",
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
}
],
"root.Phone_Numbers": [
{
"__id": "ba1560de-9c4c-4886-b4ca-684e0a7e5df0",
"__ref__root": "662783f7-b1a0-4e8c-9de9-f3a72a896d4c",
"type": "home",
"number": "555-1234"
},
{
"__id": "f1032025-6c7d-4341-8e6a-f0dce2374388",
"__ref__root": "662783f7-b1a0-4e8c-9de9-f3a72a896d4c",
"type": "work",
"number": "555-5678"
}
]
}
The result is always a dictionary where each key is a reference to the original dictionary.
- In this example, the original root document is identified by the token “
root” (the root key) and the "address" sub-dictionary is identified by “root.address”.
Each dictionary value is always a list. See below for more examples with more than one element in lists.
Each sub-dictionnary have:
- an unique field named "
__id" (like a primary key) - except for root, a "
__ref__root" who contains the "__id" value of parent dictionnary;- the "
root" token in "__ref__root" field name is directly a reference to the global result dictionnary.
- the "
Documentation
Basic usages
Empty dictionnary
>>> DictFlat().flat({})
Result:
{}
Simple dictionnary
How: Use init function "root_key" parameter.
root_key parameter signature:
str
Example:
DictFlat(
root_key="rk"
).flat(
{
"a": 1
}
)
Result:
{
"rk": [
{
"__id": "i_1",
"a": 1
}
]
}
Nested dictionnaries
2 levels
Example:
DictFlat(
root_key="rk"
).flat(
d={
"name": "John",
"address": {
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
},
"birthdate": "10/06/1976 01:10:35"
}
)
Result:
{
"rk": [
{
"__id": "i_1",
"birthdate": "10/06/1976 01:10:35",
"name": "John"
}
],
"rk.address": [
{
"__id": "i_2",
"__ref__rk": "r_3",
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
}
]
}
3 levels
Example:
DictFlat(
root_key="rk"
).flat(
d={
"name": "John",
"birth": {
"address": {
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
},
"date": "10/06/1976 01:10:35"
}
}
)
Result:
{
"rk": [
{
"__id": "i_1",
"name": "John"
}
],
"rk.birth": [
{
"__id": "i_2",
"__ref__rk": "r_3",
"date": "10/06/1976 01:10:35",
}
],
"rk.birth.address": [
{
"__id": "i_4",
"__ref__rk.birth": "r_5",
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
}
]
}
Use your own function to generate ids
How: Use init function "fct_build_id" parameter.
fct_build_id parameter signature:
def fct_name(d: Dict, path: str) -> str
By default, the "uuid4" function from "uuid" Python standard module is used.
In this example the function fct_build_id is used to generate ids.
Example:
DictFlat(
root_key="rk",
fct_build_id=fct_build_id
).flat(
d={
"name": "John",
"pers_id": 12,
"birth": {
"address": {
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
},
"date": "10/06/1976 01:10:35"
}
}
)
Result:
{
"rk": [
{
"__id": "2a02485bc672ee47",
"pers_id": 12,
"name": "John"
}
],
"rk.birth": [
{
"__id": "034b3cd2487b9d17",
"__ref__rk": "2a02485bc672ee47",
"date": "10/06/1976 01:10:35",
}
],
"rk.birth.address": [
{
"__id": "4f49da4f0b4df789",
"__ref__rk.birth": "034b3cd2487b9d17",
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
}
]
}
Change/Replace value
How: Use init function "change" parameter.
change parameter signature:
Optional[Dict[str, Callable]]
By default, no values are modified.
The dictionnary key is the future field name.
The Callable dictionnary value signature is:
def fct_name(fieldname: str, value: Any) -> Any:
Change a string by another string
Example with a date in a string to another string:
DictFlat(
root_key="rk",
fct_build_id=fct_build_id,
change={
"rk.birth.date": fix_date,
}
).flat(
d={
"name": "John",
"pers_id": 12,
"birth": {
"address": {
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
},
"date": "10/06/1976 01:10:35"
}
}
)
Result:
{
"rk": [
{
"__id": "2a02485bc672ee47",
"pers_id": 12,
"name": "John"
}
],
"rk.birth": [
{
"__id": "034b3cd2487b9d17",
"__ref__rk": "2a02485bc672ee47",
"date": "1976-06-10T01:10:35",
}
],
"rk.birth.address": [
{
"__id": "4f49da4f0b4df789",
"__ref__rk.birth": "034b3cd2487b9d17",
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
}
]
}
Change a string by a dictionnary
Example with a date in a string to a dictionnary where date and time are separated:
DictFlat(
root_key="rk",
fct_build_id=fct_build_id,
change={
"rk.birth.date": date2dict,
}
).flat(
d={
"name": "John",
"pers_id": 12,
"birth": {
"address": {
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
},
"date": "10/06/1976 01:10:35"
}
}
)
Result:
assert df == {
"rk": [
{
"__id": "2a02485bc672ee47",
"pers_id": 12,
"name": "John"
}
],
"rk.birth": [
{
"__id": "034b3cd2487b9d17",
"__ref__rk": "2a02485bc672ee47",
}
],
"rk.birth.date": [
{
"__id": "71d9d6cb90bcd168",
"__ref__rk.birth": "034b3cd2487b9d17",
"date": "1976-06-10",
"time": "01:10:35",
}
],
"rk.birth.address": [
{
"__id": "4f49da4f0b4df789",
"__ref__rk.birth": "034b3cd2487b9d17",
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
}
]
}
Drop fields (by name)
How: Use init function "drop" parameter.
drop parameter signature:
Optional[List[str]]
By default, no values are dropped.
Elements list are the future field names.
Example:
DictFlat(
root_key="rk",
fct_build_id=fct_build_id,
change={
"rk.birth.date": fix_date,
},
drop=[
"rk.birth.address.state",
]
).flat(
d={
"name": "John",
"pers_id": 12,
"birth": {
"address": {
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
},
"date": "10/06/1976 01:10:35"
}
}
)
Result:
{
"rk": [
{
"__id": "2a02485bc672ee47",
"pers_id": 12,
"name": "John"
}
],
"rk.birth": [
{
"__id": "034b3cd2487b9d17",
"__ref__rk": "2a02485bc672ee47",
"date": "1976-06-10T01:10:35",
}
],
"rk.birth.address": [
{
"__id": "4f49da4f0b4df789",
"__ref__rk.birth": "034b3cd2487b9d17",
"street": "123 Main St",
"city": "Anytown"
}
]
}
Rename/Change field names
How: Use init function "rename" parameter.
rename parameter signature:
Optional[Dict[str, Union[str, Callable]]]
By default, no field names values are modified.
The dictionnary key is the first version of the future field name. The dictionnary value is the final field name or a function to genrate the new field name.
If the dictionnary key is a "Callable", the signature is:
def fct_name(s: str) -> str
IMPORTANT
If you use "change" parameter and "rename" parameter, use the final field name in "rename" dictionnary value as "change" dictionnary key.
Rename field name by name
Example:
DictFlat(
root_key="rk",
fct_build_id=fct_build_id,
change={
"rk.birth.date_dict": date2dict,
},
rename={
"rk.birth.date": "rk.birth.date_dict",
"PersId": "pers_id",
}
).flat(
d={
"name": "John",
"PersId": 12,
"birth": {
"address": {
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
},
"date": "10/06/1976 01:10:35"
}
}
)
Result:
{
"rk": [
{
"__id": "2a02485bc672ee47",
"pers_id": 12,
"name": "John"
}
],
"rk.birth": [
{
"__id": "034b3cd2487b9d17",
"__ref__rk": "2a02485bc672ee47",
}
],
"rk.birth.date_dict": [
{
"__id": "71d9d6cb90bcd168",
"__ref__rk.birth": "034b3cd2487b9d17",
"date": "1976-06-10",
"time": "01:10:35",
}
],
"rk.birth.address": [
{
"__id": "4f49da4f0b4df789",
"__ref__rk.birth": "034b3cd2487b9d17",
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
}
]
}
Rename ALL field names
You could rename all fields using the "RENAME_ALL" special key and un function to do this.
The "Callable" dictionnary key function signature is:
def fct_name(s: str) -> str
IMPORTANT
You could use "RENAME_ALL" key and field name keys. "RENAME_ALL" key is allway use BEFORE field name keys
In this example, the function "str_2_snakecase" is called before the other rename key.
Example:
DictFlat(
root_key="rk",
fct_build_id=fct_build_id,
change={
"rk.birth.date_dict": date2dict,
},
rename={
RENAME_ALL: str_2_snakecase,
"rk.birth.date": "rk.birth.date_dict",
}
).flat(
d={
"Name": "John",
"PersId": 12,
"Birth": {
"Address": {
"Street": "123 Main St",
"City": "Anytown",
"State": "CA"
},
"Date": "10/06/1976 01:10:35"
}
}
)
Result:
{
"rk": [
{
"__id": "2a02485bc672ee47",
"pers_id": 12,
"name": "John"
}
],
"rk.birth": [
{
"__id": "034b3cd2487b9d17",
"__ref__rk": "2a02485bc672ee47",
}
],
"rk.birth.date_dict": [
{
"__id": "71d9d6cb90bcd168",
"__ref__rk.birth": "034b3cd2487b9d17",
"date": "1976-06-10",
"time": "01:10:35",
}
],
"rk.birth.address": [
{
"__id": "4f49da4f0b4df789",
"__ref__rk.birth": "034b3cd2487b9d17",
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
}
]
}
Lists
Each element of a list a transformed as a dictionnary of a same type.
List of dictionaries
By default, no element are added in each dictionnary.
Example:
DictFlat(
root_key="rk",
fct_build_id=fct_build_id,
change={
"rk.birth.date_dict": date2dict,
},
rename={
"rk.birth.date": "rk.birth.date_dict",
}
).flat(
d={
"name": "John",
"pers_id": 12,
"birth": {
"address": {
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
},
"date": "10/06/1976 01:10:35"
},
"Phone_Numbers": [
{"type": "home", "number": "555-1234"},
{"type": "work", "number": "555-5678"},
],
}
)
Result:
{
"rk": [
{
"__id": "2a02485bc672ee47",
"pers_id": 12,
"name": "John"
}
],
"rk.birth": [
{
"__id": "034b3cd2487b9d17",
"__ref__rk": "2a02485bc672ee47",
}
],
"rk.birth.date_dict": [
{
"__id": "71d9d6cb90bcd168",
"__ref__rk.birth": "034b3cd2487b9d17",
"date": "1976-06-10",
"time": "01:10:35",
}
],
"rk.birth.address": [
{
"__id": "4f49da4f0b4df789",
"__ref__rk.birth": "034b3cd2487b9d17",
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
}
],
"rk.Phone_Numbers": [
{
"__id": "5d1765d47e80b6d3",
"__ref__rk": "2a02485bc672ee47",
"type": "home",
"number": "555-1234"
},
{
"__id": "87d897df197fbdc7",
"__ref__rk": "2a02485bc672ee47",
"type": "work",
"number": "555-5678"
},
],
}
Add counter field in each list element
How: Use init function "list_2_object" parameter.
list_2_object parameter signature:
Optional[Dict[str, Dict]]
By default, no fields are added.
The dictionnary key is the first version of the future field name. The dictionnary value is sub-dictionnary for parametrize the job:
- The key "
counter_field" contains the field name (a "str") for the counter;- Default value is "
idx".
- Default value is "
- The key "
starts_at" contains the counter start value (a "int").- Default value is
1.
- Default value is
Example:
DictFlat(
root_key="rk",
fct_build_id=fct_build_id,
change={
"rk.birth.date_dict": date2dict,
},
rename={
"rk.birth.date": "rk.birth.date_dict",
},
list_2_object={
"rk.Phone_Numbers": {
"counter_field": "count",
"starts_at": 0
}
}
).flat(
d={
"name": "John",
"pers_id": 12,
"birth": {
"address": {
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
},
"date": "10/06/1976 01:10:35"
},
"Phone_Numbers": [
{"type": "home", "number": "555-1234"},
{"type": "work", "number": "555-5678"},
],
}
)
Result:
{
"rk": [
{
"__id": "2a02485bc672ee47",
"pers_id": 12,
"name": "John"
}
],
"rk.birth": [
{
"__id": "034b3cd2487b9d17",
"__ref__rk": "2a02485bc672ee47",
}
],
"rk.birth.date_dict": [
{
"__id": "71d9d6cb90bcd168",
"__ref__rk.birth": "034b3cd2487b9d17",
"date": "1976-06-10",
"time": "01:10:35",
}
],
"rk.birth.address": [
{
"__id": "4f49da4f0b4df789",
"__ref__rk.birth": "034b3cd2487b9d17",
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
}
],
"rk.Phone_Numbers": [
{
"__id": "30fb5bc71274e531",
"__ref__rk": "2a02485bc672ee47",
"count": 0,
"type": "home",
"number": "555-1234"
},
{
"__id": "6b74835b56e08367",
"__ref__rk": "2a02485bc672ee47",
"count": 1,
"type": "work",
"number": "555-5678"
},
],
}
List of non-dictionary elements
If the list do not contains dictionnary elements, you could specify the name of the future key with the suffix ".__inner".
Example:
DictFlat(
root_key="rk",
fct_build_id=fct_build_id,
change={
"rk.birth.date_dict": date2dict,
},
rename={
RENAME_ALL: str_2_snakecase,
"rk.birth.date": "rk.birth.date_dict",
"rk.phone_numbers.__inner": "number",
}
).flat(
d={
"name": "John",
"pers_id": 12,
"birth": {
"address": {
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
},
"date": "10/06/1976 01:10:35"
},
"Phone_Numbers": [
"555-1234",
"555-5678",
],
}
)
Result:
{
"rk": [
{
"__id": "2a02485bc672ee47",
"pers_id": 12,
"name": "John"
}
],
"rk.birth": [
{
"__id": "034b3cd2487b9d17",
"__ref__rk": "2a02485bc672ee47",
}
],
"rk.birth.date_dict": [
{
"__id": "71d9d6cb90bcd168",
"__ref__rk.birth": "034b3cd2487b9d17",
"date": "1976-06-10",
"time": "01:10:35",
}
],
"rk.birth.address": [
{
"__id": "4f49da4f0b4df789",
"__ref__rk.birth": "034b3cd2487b9d17",
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
}
],
"rk.phone_numbers": [
{
"__id": "24886b1e9942f612",
"__ref__rk": "2a02485bc672ee47",
"number": "555-1234"
},
{
"__id": "a98b1c4fa2e2a2b5",
"__ref__rk": "2a02485bc672ee47",
"number": "555-5678"
},
],
}
Squash dictionnaries
When you have a dictionary of dictionaries and want to have just one.
How: Use init function "dict_of_dicts_2_dict" parameter.
dict_of_dicts_2_dict parameter signature:
Optional[Dict[str, Dict]]
The key is the future dictionnary name The value is the definition of treatment:
- "
sep": The separator between the key of first data dictionnary and the key of second data dictionnary (default value is a dot ".")- You could change the default value for all separators with "
sep" init function.
- You could change the default value for all separators with "
- "
reverse": To change the order of the keys on each side of the separator (default value isFalse)
Example:
From:
{
"miracles": {
"first": {
"k": "one",
"e": "e-one"
},
"second": {
"k": "two"
},
"third": "three"
}
}
To:
{
"rk.miracles": [
{
"__id": "041102055056a3a8",
"__ref__rk": "2a02485bc672ee47",
"first/k": "one",
"first/e": "e-one",
"second/k": "two",
"third": "three",
},
]
}
DictFlat(
root_key="rk",
fct_build_id=fct_build_id,
change={
"rk.birth.date_dict": date2dict,
},
rename={
"rk.birth.date": "rk.birth.date_dict",
},
dict_of_dicts_2_dict={
"rk.miracles": {
"reverse": False,
"sep": "/"
}
}
).flat(
d={
"name": "John",
"pers_id": 12,
"birth": {
"address": {
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
},
"date": "10/06/1976 01:10:35"
},
"miracles": {
"first": {
"k": "one",
"e": "e-one"
},
"second": {
"k": "two"
},
"third": "three"
}
}
)
Result:
{
"rk": [
{
"__id": "2a02485bc672ee47",
"pers_id": 12,
"name": "John"
}
],
"rk.birth": [
{
"__id": "034b3cd2487b9d17",
"__ref__rk": "2a02485bc672ee47",
}
],
"rk.birth.date_dict": [
{
"__id": "71d9d6cb90bcd168",
"__ref__rk.birth": "034b3cd2487b9d17",
"date": "1976-06-10",
"time": "01:10:35",
}
],
"rk.birth.address": [
{
"__id": "4f49da4f0b4df789",
"__ref__rk.birth": "034b3cd2487b9d17",
"street": "123 Main St",
"city": "Anytown",
"state": "CA"
}
],
"rk.miracles": [
{
"__id": "041102055056a3a8",
"__ref__rk": "2a02485bc672ee47",
"first/k": "one",
"first/e": "e-one",
"second/k": "two",
"third": "three",
},
],
}
Nested dictionnaries with simple key names
When you don't want long key names.
How: Use init function "simple_keys" parameter.
simple_keys parameter signature:
bool
Default value is False.
!Warning!
If you use this parameter with True value. The result may contains some unexpected values. If you have input with two or more sub-dictionnaries with the same name but in different paths, the output contains only one general key. See second example
Simple
Example:
DictFlat(
root_key="rk",
simple_keys=True
).flat(
d={
"name": "John",
"birth": {
"address": {
"street": {
"number": "123",
"road": "Main St"
},
"city": "Anytown",
"state": "CA"
},
"date": "10/06/1976 01:10:35"
}
}
)
Result:
{
"rk": [
{
"__id": "i_1",
"name": "John"
}
],
"birth": [
{
"__id": "i_2",
"__ref__rk": "r_3",
"date": "10/06/1976 01:10:35",
}
],
"address": [
{
"__id": "i_4",
"__ref__birth": "r_5",
"city": "Anytown",
"state": "CA"
}
],
"street": [
{
"__id": "i_6",
"__ref__address": "r_7",
"number": "123",
"road": "Main St",
},
]
}
Unexpected result
Example:
DictFlat(
root_key="rk",
simple_keys=True
).flat(
d={
"name": "John",
"birth": {
"address": {
"street": {
"number": "123",
"road": "Main St"
},
"city": "Anytown",
"state": "CA"
},
"date": "10/06/1976 01:10:35"
},
"street": {
"other": "abc"
}
}
)
Result:
{
"rk": [
{
"__id": "i_1",
"name": "John"
}
],
"birth": [
{
"__id": "i_2",
"__ref__rk": "r_3",
"date": "10/06/1976 01:10:35",
}
],
"address": [
{
"__id": "i_4",
"__ref__birth": "r_5",
"city": "Anytown",
"state": "CA"
}
],
"street": [
{
"__id": "i_6",
"__ref__address": "r_7",
"number": "123",
"road": "Main St",
},
{
"__id": "i_8",
"__ref__rk": "r_9",
"other": "abc",
},
]
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file dictflat-0.1.5.tar.gz.
File metadata
- Download URL: dictflat-0.1.5.tar.gz
- Upload date:
- Size: 10.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/2.1.3 CPython/3.13.3 Linux/6.14.9-200.fc41.x86_64
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
7f9ec5f5bb187fe8cb518506992cf08cde37e74e84020734e797a3dcde26a6d4
|
|
| MD5 |
0319df8aea2ce1533d7e508406cc0001
|
|
| BLAKE2b-256 |
4ddff82a0d97dfa5205a928d18019ab6fe391936bfb0ce35fe1d81415a31e25c
|
File details
Details for the file dictflat-0.1.5-py3-none-any.whl.
File metadata
- Download URL: dictflat-0.1.5-py3-none-any.whl
- Upload date:
- Size: 8.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/2.1.3 CPython/3.13.3 Linux/6.14.9-200.fc41.x86_64
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8a7ee96a9cf84b2bb20a3148d290da28ccd0da7e8fec6964f97bdb026ded8b9c
|
|
| MD5 |
3583bb9578755ffd0d45c91f16e14362
|
|
| BLAKE2b-256 |
a49c922bc8a9b402b08e32576847eed96dc1a9686aa7e7014a1376cb5b7a8f9c
|