[Python] Convert general-purpose container and class to each other

Rough summary

It's been a while, but it's a sequel to the article I wrote earlier. The one I made this time is uploaded to github. The one that can be installed with pip is for myself.

What I wanted to do

  1. I tried to process the data structure received by json and put it in DynamoDB with another data structure.
  2. Tired of more code that directly manipulates dicts and lists. It makes me feel object-oriented, wanting to give behavior to meaningful data itself.
  3. Then, let's convert it to class. Let's convert the processing result class to dict.

What I actually did

  1. Convert "structure made of general-purpose container such as dict and list" to "arbitrary class structure". And vice versa. [^ generic-conteiners]
  2. I made a mapping to define the conversion rule with dict, and made a class to convert according to it.
  3. A usage example will be introduced in the unit test.

Put the actual code on github.

Make it right away

I made something similar before, but the code has changed and time has passed, so I will write from 1 without breaking ..

image

For example, suppose you have the following dict as the source of conversion. It is a structure that includes dict in dict.

Source of conversion source

src_dict = {
    'value' : 'AAA',
    'nested' : {
        'nested_value' : 'BBB'
    }
}

Suppose you want to convert this to a class like this: I want to have an instance of NestedClass in self.nested of TastClass.

Destination class

#Class 1
class TestClass():
    def test_method(self):
        return 'assigned value: ' + self.value
#Class 2-Image hanging in part 1
class NestedTestClass:
    def test_method(self):
        return 'nested assigned value: ' + self.nested_value

Information that connects two data

In order to perform the above conversion, I think that it is necessary to map which element of the container is converted to which class, so I thought about this type of dict.

mapping = {
    '<MAPPING_ROOT>' : TestClass, #Because the top has no name'<MAPPING_ROOT>'To
    'nested' : NestedClass
}

The mapping of [class / dict key]: [function] is expressed by dict. The dict itself pointed to by src_dict has no name, so use<MAPPING_ROOT>as a key instead. The constructor is specified as a function. Items not included in mapping, such assrc_dict ['value']in this example, are set as they are in the conversion destination.

How to use

I want to use it like this.

usage


#Take a mapping in the constructor
converter = ObjectConverter(mapping=mapping)
#Pass the conversion source data to the conversion method
converted_class = converter.convert(src_dict)
#Call a method of the converted class
converted_class.test_method()

Implementation

I made it like this.

I made it as converter.py, but it's long. Fold it up.

converter.py



class ObjectConverter:
    #Receive mapping definition at generation
    def __init__(self, *, mapping):
        self.mapping = mapping

    #Transform call method
    def convert(self, src):
        #Top element is mapping'<root>'Premise that always matches
        return self._convert_value('<MAPPING_ROOT>', self.mapping['<MAPPING_ROOT>'], src)

    #Determine the processing method according to the value
    def _convert_value(self, key, func, value):
        #In the case of a list, all the elements are converted with func
        if isinstance(value, (list, tuple)):
            return self._convert_sequence(key, func, value)

        #In the case of dict, retrieve the key and value as they are
        if isinstance(value, dict):
            return self._convert_dict(key, func, value)

        #For class__dict__And treat it as a dict
        if isinstance(value, object) and hasattr(value, '__dict__'):
            return self._convert_dict(key, func, value.__dict__)

        #If none of the above applies, return it as is
        return value

    #Convert the contents of the dict
    def _convert_dict(self, key, func, src):
        # _call_Fill the object created by function
        return self._assign_dict(self._call_function(key, func, src), key, src)

    #Creation of the object specified by mapping
    def _call_function(self, key, func, src):
        return func()

    #Take out the contents of the dict and apply it
    def _assign_dict(self, dest, key, src):

        for srcKey, value in src.items():

            #key is defined in the mapping
            if srcKey in self.mapping:
                func = self.mapping[srcKey]
                #Execute the mapped function and set the result
                self._set_value(dest, srcKey, self._convert_value(srcKey, func, value))

            #If the key is not in the mapping definition, set it as it is
            else:
                #Even if there is a mapping defined value in the value passed here, it will be ignored.
                self._set_value(dest, srcKey, value)

        #The state where the contents of src are reflected in created
        return dest

    #List processing
    def _convert_sequence(self, key, func, sequence):
        current = []
        for value in sequence:
            current.append(self._convert_value(key, func, value))
        return current

    #Value setter for both dict and class
    def _set_value(self, dest, key, value):
        if isinstance(dest, dict):
            dest[key] = value
        else:
            setattr(dest, key, value)

    #Utility method to get an instance for dict conversion
    #
    @classmethod
    def dict_converter(cls, mapping, *, dict_object=dict):
        reverse_mapping = {}

        #Make all mapping destinations dict
        for key in mapping.keys():
            reverse_mapping[key] = dict_object

        #Instance for converting to dict
        return ObjectConverter(mapping=reverse_mapping)

Rough commentary

** See source for full specifications. ** But nothing happens, it just scans the value that matches the key contained in the mapping. Roughly speaking, I am doing this.

Touch the contents of the class

By extracting __dict__ when a class is found, all you have to do is scan the dict. I'd like to avoid touching __dict__ if possible, but this time I'll avoid the effort of using other methods. Unless you are dealing with a special class, there should be no problem. [^ dont-touch-this]

Case that does not respond to the key

The value not included in mapping is set to the conversion destination object as it is, but even if the value at this time is dict and there is a value with the name included in mapping in it, mapping processing is performed. It will not be. This is "processed dict included in mapping or included in mapping ue I don't like the case of "dict processing", and I think that the data structure that requires such conversion is not very beautiful.

class to dict conversion

For cases where you want to do a class to dict conversion after a dict to class conversion to get it back to its original form, we have a class method dict_converter to make it easy to get a reverse converter. It's easy because all the conversion destinations on mapping are set to dict. [^ not-dict-something]

Unit test that also explains how to use

It's long, so fold it.

test_objectconverter.py


import unittest
import json
from objectonverter import ObjectConverter

#Test class 1
class TestClass():
    def test_method(self):
        return 'TestObject.test_method'

#Test class part 2
class NestedTestClass:
    def test_method(self):
        return 'NestedObject.test_method'


class ClassConverterTest(unittest.TestCase):

    #Just set the properties of the root class
    def test_object_convert(self):
        dict_data = {
            'value1' : 'string value 1'
        }

        converter = ObjectConverter(mapping={'<MAPPING_ROOT>' : TestClass})
        result = converter.convert(dict_data)
        self.assertEqual(result.value1, 'string value 1')

        #Try to call the method of the generated class
        self.assertEqual(result.test_method(), 'TestObject.test_method')

    #Generate a nested class
    def test_nested_object(self):
        #dict that maps json keys and classes
        object_mapping = {
            '<MAPPING_ROOT>' : TestClass,
            'nested' : NestedTestClass
        }
        #Source of origin
        dict_data = {
            'value1' : 'string value 1',
            'nested' : {
                'value' : 'nested value 1'
            }
       }

        converter = ObjectConverter(mapping=object_mapping)
        result = converter.convert(dict_data)
        self.assertEqual(result.value1, 'string value 1')

        self.assertIsInstance(result.nested, NestedTestClass)
        self.assertEqual(result.nested.value, 'nested value 1')

    #Just a dict if you don't specify a mapping
    def test_nested_dict(self):
        object_mapping = {
            '<MAPPING_ROOT>' : TestClass
        }

        #Source of origin
        dict_data = {
            'value1' : 'string value 1',
            'nested' : {
                'value' : 'nested value 1'
            }
        }

        converter = ObjectConverter(mapping = object_mapping)
        result = converter.convert(dict_data)
        self.assertEqual(result.value1, 'string value 1')
        self.assertIsInstance(result.nested, dict)
        self.assertEqual(result.nested['value'], 'nested value 1')

    #List processing
    def test_sequence(self):
        mapping = {
            '<MAPPING_ROOT>' : TestClass,
            'nestedObjects' : NestedTestClass,
        }
        source_dict = {
            "value1" : "string value 1",
            "nestedObjects" : [
                {'value' : '0'},
                {'value' : '1'},
                {'value' : '2'},
            ]
        }

        converter = ObjectConverter(mapping=mapping)
        result = converter.convert(source_dict)
        self.assertEqual(result.value1, 'string value 1')
        self.assertEqual(len(result.nestedObjects), 3)

        for i in range(3):
            self.assertIsInstance(result.nestedObjects[i], NestedTestClass)
            self.assertEqual(result.nestedObjects[i].value, str(i))

    #If the root element itself is a list
    def test_root_sequence(self):
        object_mapping = {
            '<MAPPING_ROOT>' : TestClass,
        }

        source_list = [
            {'value' : '0'},
            {'value' : '1'},
            {'value' : '2'},
        ]

        converter = ObjectConverter(mapping=object_mapping)
        result = converter.convert(source_list)

        self.assertIsInstance(result, list)
        self.assertEqual(len(result), 3)

        for i in range(3):
            self.assertIsInstance(result[i], TestClass)
            self.assertEqual(result[i].value, str(i))

    # json -> class -> json
    def test_json_to_class_to_json(self):
        #Function used for mutual conversion from class to json
        def default_method(item):
            if isinstance(item, object) and hasattr(item, '__dict__'):
                return item.__dict__
            else:
                raise TypeError

        #dict that maps json keys and classes
        object_mapping = {
            '<MAPPING_ROOT>' : TestClass,
            'nested' : NestedTestClass
        }
        #Source of origin-In one line for convenience of comparison
        string_data = '{"value1": "string value 1", "nested": {"value": "nested value 1"}}'
        dict_data = json.loads(string_data)

        converter = ObjectConverter(mapping=object_mapping)
        result = converter.convert(dict_data)
        dump_string = json.dumps(result, default=default_method)
        self.assertEqual(dump_string, string_data)

        #The result is the same even if it is converted again
        result = converter.convert(json.loads(dump_string))
        self.assertEqual(result.value1, 'string value 1')
        self.assertIsInstance(result.nested, NestedTestClass)
        self.assertEqual(result.nested.value, 'nested value 1')

    #conversion->Inverse conversion
    def test_reverse_convert(self):
        dict_data = {
            'value1' : 'string value 1'
        }
        mapping = {'<MAPPING_ROOT>' : TestClass}

        converter = ObjectConverter(mapping=mapping)
        result = converter.convert(dict_data)
        self.assertEqual(result.value1, 'string value 1')
        
        #Generate inverse conversion converter
        reverse_converter = ObjectConverter.dict_converter(mapping=mapping)
        reversed_result = reverse_converter.convert(result)
        self.assertEqual(result.value1, reversed_result['value1'])



if __name__ == '__main__':
    unittest.main()

Commentary

Well, I wonder if you can understand the basics There is a test case with the long name test_json_to_class_to_json, but this is because this class was originally very conscious of conversion to json.

Snake excuse

It's been more than half a year since the last article, but in fact I finally got a job ... I didn't have time, so it was late.

[^ generic-conteiners]: Since it is long to write "dict / list" many times below, I will write it as a general-purpose container. By the way, a "class" is also an "instance of a class" to be exact, but it is a "class" because it is long. [^ json-to-class]: The first thing I was thinking about was how to handle json, so I was trapped by json. As I chase things, I realize that I was actually looking for very simple results. It's a “something” flow, but you have to think more carefully. [^ dont-touch-this]: It's easy, so I'll just do it, but __dict __ is like a back door, and when you retrieve the value from __dict __, __getattribute__ may not be called, so the class It may behave differently than it was originally intended. I won't think about that here. [^ not-dict-something]: I wrote it to support types other than Ichiou dict, but I can't think of any use for it.

Recommended Posts

[Python] Convert general-purpose container and class to each other
[Python learning part 3] Convert pandas DataFrame, Series, and standard List to each other
Try converting latitude / longitude and world coordinates to each other with python
How to convert SVG to PDF and PNG [Python]
[Python] Convert decimal numbers to binary numbers, octal numbers, and hexadecimal numbers
Convert / return class object to JSON format in Python
Python: Class and instance variables
[python] Convert date to string
Convert numpy int64 to python int
[Python] Convert list to Pandas [Pandas]
Convert Scratch project to Python
Python class variables and instance variables
[Python] Convert Shift_JIS to UTF-8
Convert python 3.x code to python 2.x
[Python] How to play with class variables with decorator and metaclass
How to convert Youtube to mp3 and download it super-safely [Python]
Convert video to black and white with ffmpeg + python + opencv
Determine the date and time format in Python and convert to Unixtime
How to write a Python class
Python 3.6 on Windows ... and to Xamarin.
[Introduction to Python3 Day 1] Programming and Python
Convert markdown to PDF in Python
Python class definitions and instance handling
Workflow to convert formula (image) to python
Convert list to DataFrame with python
[Python] Road to snake charmer (3) Python class
Read big endian binary in Python and convert it to ndarray
Python> list> Convert double list to single list
[Python] Convert natural numbers to ordinal numbers
perl objects and python class part 1.
Python logging and dump to json
Convert decimal numbers to n-ary numbers [python]
Convert the result of python optparse to dict and utilize it
Selenium and python to open google
I was addicted to confusing class variables and instance variables in Python
Python> tuple> Convert double tuple to single tuple
Read CSV file with Python and convert it to DataFrame as it is
[Python] How to convert db file to csv
How to package and distribute Python scripts
From Python to using MeCab (and CaboCha)
Convert memo at once with Python 2to3
Convert Python> two value sequence to dictionary
How to install and use pandas_datareader [Python]
[Python] How to make a class iterable
Bind methods to Python classes and instances
[Python] How to convert a 2D list to a 1D list
How to convert Python to an exe file
[Python] Convert csv file delimiters to tab delimiters
Bind to class to read and write YAML
Convert psd file to png in Python
Convert Excel data to JSON with python
Convert Hiragana to Romaji with Python (Beta)
Fractal to make and play with Python
Convert from katakana to vowel kana [python]
Convert FX 1-minute data to 5-minute data with Python
Python basic operation 3rd: Object-oriented and class
python> Convert tuple to list> aList = list (pi_tuple)
[Python] Difference between class method and static method
Porting and modifying doublet-solver from python2 to python3.
Read Python csv and export to txt
python: How to use locals () and globals ()