memory profiler 를 활용해서 메모리 사용량 확인하기

목차

패키지 설치

pip install memory_profiler

실행방법

memory profiler를 작동하려면 다음과 같이 한다.

python script가 종료되어야 가능하다.

python -m memory_profiler test.py

memory profiler를 logging 하려면 다음과 같이 하면 된다.

logging은 다음과 같이 하면 된다.

python -m memory_profiler test.py > ./log.txt

@profile로 지정하여야 그 부분에 대해서 확인이 된다!

# imports 
from memory_profiler import profile 
import requests 
  
class BaseExtractor: 
    # decorator which specifies which 
    # function to monitor 
    @profile
    def parse_list(self, array): 
        
        # create a file object 
        f = open('words.txt', 'w') 
        for word in array: 
              
            # writing words to file 
            f.writelines(word) 
      
    # decorator which specifies which  
    # function to monitor 
    @profile
    def parse_url(self, url): 
        
        # fetches the response 
        response = requests.get(url).text 
        with open('url.txt', 'w') as f: 
              
            # writing response to file 
            f.writelines(response)
  
if __name__ == "__main__": 
      
    # url for word list (huge) 
    url = 'https://raw.githubusercontent.com/dwyl/english-words/master/words.txt'
      
    # word list in array 
    array = ['one', 'two', 'three', 'four', 'five'] 
      
    # initializing BaseExtractor object 
    extractor = BaseExtractor() 
      
    # calling parse_url function 
    extractor.parse_url(url) 
      
    # calling pasrse_list function 
    extractor.parse_list(array)

 

결과 시각화

 mprof run julia_example.py

??.dat 파일이 생성된다고 함.

 mprof plot -o image.png --backend agg

logging 파일 생성하는 코드

from memory_profiler import profile
import logging

# create logger
logger = logging.getLogger('memory_profile_log')
logger.setLevel(logging.DEBUG)

# create file handler which logs even debug messages
fh = logging.FileHandler("memory_profile.log")
fh.setLevel(logging.DEBUG)

# create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)

# add the handlers to the logger
logger.addHandler(fh)

from memory_profiler import LogFile
import sys
sys.stdout = LogFile('memory_profile_log', reportIncrementFlag=False)

 

 

 

결과물 로깅 하는 방법

profile에 stream을 생성하여, logging 하기

from memory_profiler import profile

f=open('hi.txt','w+')

@profile(stream=f)
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

• mprof run: running an executable, recording memory usage
• mprof plot: plotting one the recorded memory usage (by default, the last one)
• mprof list: listing all recorded memory usage files in a user-friendly way.
• mprof clean: removing all recorded memory usage files.
• mprof rm: removing specific recorded memory usage files

 

Plot settings

By default, the command line call is set as the graph title. If you wish to customize it, you can use the -t option to manually set the figure title.

mprof plot -t 'Recorded memory usage'

You can also hide the function timestamps using the n flag, such as

mprof plot -n

Trend lines and its numeric slope can be plotted using the s flag, such as

mprof plot -s

 

>>> # define a simple function
>>> def f(a, n=100):
    ...     import time
    ...     time.sleep(2)
    ...     b = [a] * n
    ...     time.sleep(1)
    ...     return b
    ...
>>> from memory_profiler import memory_usage
>>> memory_usage((f, (1,), {'n' : int(1e6)}))

 

참고

https://pythonrepo.com/repo/pythonprofilers-memory_profiler-python-monitoring

Monitor Memory usage of Python code | PythonRepo

https://coderzcolumn.com/tutorials/python/how-to-profile-memory-usage-in-python-using-memory-profiler

How to Profile Memory Usage in Python using memory_profiler? by Sunny Solanki