MemoryError with django

Question

I have a django app that loads a lot of data into a sqlite3 database file.
We are talking about millions of entries, entered in one function that at some point of its execution is throwing:

  File "/root/codebase/lib/python2.6/site-packages/Django-1.3.1-py2.6.egg/django/db/transaction.py", line 219, in inner
    self.__exit__(*sys.exc_info())
  File "/root/codebase/lib/python2.6/site-packages/Django-1.3.1-py2.6.egg/django/db/transaction.py", line 207, in __exit__
    self.exiting(exc_value, self.using)
  File "/root/codebase/lib/python2.6/site-packages/Django-1.3.1-py2.6.egg/django/db/transaction.py", line 286, in exiting
    leave_transaction_management(using=using)
  File "/root/codebase/lib/python2.6/site-packages/Django-1.3.1-py2.6.egg/django/db/transaction.py", line 56, in leave_transaction_management
    connection.leave_transaction_management()
  File "/root/codebase/lib/python2.6/site-packages/Django-1.3.1-py2.6.egg/django/db/backends/__init__.py", line 114, in leave_transaction_management
    self.rollback()
MemoryError

I examined my function and it is NOT leaking memory. What else can be done here?

Answer 1

from https://docs.djangoproject.com/en/dev/faq/models/#why-is-django-leaking-memory

Django isn't known to leak memory. If you find your Django processes are allocating more and more memory, with no sign of releasing it, check to make sure your DEBUG setting is set to False. If DEBUG is True, then Django saves a copy of every SQL statement it has executed.

(The queries are saved in django.db.connection.queries. See How can I see the raw SQL queries Django is running?.)

To fix the problem, set DEBUG to False.

If you need to clear the query list manually at any point in your functions, just call reset_queries(), like this:

from django import db
db.reset_queries()

Answer 2

SQLite is an in-memory database. It's stored on your hard drive for posterity, but it's loaded in entirety into memory to interact with it. That means it's inherently limited by the memory resources of your system. It was never designed to be a full-fledged database solution, but rather a speedy database to handle data for which larger database solutions like PostgreSQL and MySQL would be overkill. It's also great for development environments, where it's easy to use, usually pre-installed, and perfectly sufficient for the amount of data used in development and testing. As a production database, though, you'd be insane to use it.

However, even if you're just in development, if you're dealing with "millions of entries", you've officially outgrown SQLite.

Answer 3

The function may not be, but if your realing with such a huge database, the sqlite library may be consuming it in order to update the table. If you need to store such huge amounts of data you may have to move away from SQLite to a fully featured database server such as MySQL.