Python has a serialization module for objects called pickle (cPickle). I think pickle is very famous. In fact, there is also a similar module called marshal. This is not very famous. It's not a persistent module, and compatibility between versions isn't guaranteed, so there's no reason to use marshal. Not surprisingly, it's not famous.
Marshal is full of drawbacks compared to pickle, but I found that marshal also has some good points, so this time I will focus on that. The good thing about marshal is "speed".
Let's check the export and import speeds immediately. This time, the speeds were compared between "cPickle" and "marshal".
First, write it out.
#!/usr/bin/env python
# -- coding:utf-8 -*-
import marshal
import cPickle as pickle
import time
import numpy as np
def main():
a = np.ndarray(shape=(10000, 10000))
start = time.time()
pickle.dump(a, open('output.pkl', 'wb'))
p_time = time.time() - start
start = time.time()
marshal.dump(a, open('output.msl', 'wb'))
m_time = time.time() - start
print p_time, m_time
if __name__ == '__main__':
main()
The code looks like this. I'm generating a multidimensional array and writing it out.
Let's take a look at the output.
143.123441935 5.09839010239
The left is cPickle, the right is marshal, and the unit is seconds. It made a difference more than I expected, it was unexpected.
Next is reading.
#!/usr/bin/env python
# -- coding:utf-8 -*-
import marshal
import cPickle as pickle
import time
import numpy as np
def main():
start = time.time()
a = pickle.load(open('output.pkl', 'rb'))
p_time = time.time() - start
start = time.time()
b = marshal.load(open('output.msl', 'rb'))
m_time = time.time() - start
print p_time, m_time
if __name__ == '__main__':
main()
The code is this. I just set dump to load.
Click here for results
445.698551893 1.64994597435
Similarly, the left is cPickle, the right is marshal, and the unit is seconds. Well, it's a huge difference ... I'm pretty surprised.
Yes. So it turns out that marshal is faster in speed. I haven't investigated the reason why.
However, although marshal is superior in speed, pickle that makes it permanent is overwhelmingly convenient, so if you use it, it is still pickle. Then why did you compare them? The only reason is that I was curious.
So I spotlighted marshal, which I would rarely use! That is all for the story.
Recommended Posts