diff options
| author | Yuval Adam <yuval@segmanta.com> | 2014-03-20 17:49:16 +0200 |
|---|---|---|
| committer | Yuval Adam <yuval@segmanta.com> | 2014-03-20 17:49:16 +0200 |
| commit | 75f93a3f7854a7383febd047391e2bfd4caceee0 (patch) | |
| tree | d5e634ef292c6b84acacc3982dbcaa0d300a346e | |
67 files changed, 5906 insertions, 0 deletions
diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..2e3d5a6 --- /dev/null +++ b/.gitignore @@ -0,0 +1,24 @@ +syntax:glob + +*.DS_Store +*.egg +*.egg-info +*.elc +*.gz +*.log +*.orig +*.pyc +*.swp +*.tmp +*~ +.hg/ +.tox/ +_build/ +build/ +dist/* +.hgignore +.hgtags +django +local_settings.py +setuptools* +testdb.sqlite @@ -0,0 +1,26 @@ +By order of apparition, thanks: + * Marty Alchin (S3) + * David Larlet (S3) + * Arne Brodowski (S3) + * Sebastian Serrano (S3) + * Andrew McClain (MogileFS) + * Rafal Jonca (FTP) + * Chris McCormick (S3 with Boto) + * Ivanov E. (Database) + * Ariel Núñez (packaging) + * Wim Leers (SymlinkOrCopy + patches) + * Michael Elsdörfer (Overwrite + PEP8 compatibility) + * Christian Klein (CouchDB) + * Rich Leland (Mosso Cloud Files) + * Jason Christa (patches) + * Adam Nelson (patches) + * Erik CW (S3 encryption) + * Axel Gembe (Hash path) + * Waldemar Kornewald (MongoDB) + * Russell Keith-Magee (Apache LibCloud patches) + * Jannis Leidel (S3 and GS with Boto) + * Andrei Coman (Azure) + * Chris Streeter (S3 with Boto) + +Extra thanks to Marty for adding this in Django, +you can buy his very interesting book (Pro Django). diff --git a/CHANGELOG.rst b/CHANGELOG.rst new file mode 100644 index 0000000..df3233a --- /dev/null +++ b/CHANGELOG.rst @@ -0,0 +1,110 @@ +django-storages change log +========================== + +1.1.8 (2013-03-31) +****************** + +* Fixes `#156`_ regarding date parsing, ValueError when running collectstatic +* Proper handling of boto dev version parsing +* Made SFTP URLs accessible, now uses settings.MEDIA_URL instead of sftp:// + +.. _#156: https://bitbucket.org/david/django-storages/issue/156/s3boto-backend-valueerror-time-data-thu-07 + +1.1.7 (2013-03-20) +****************** + +* Listing of huge buckets on S3 is now prevented by using the prefix argument to boto's list() method +* Initial support for Windows Azure Storage +* Switched to useing boto's parse_ts date parser getting last modified info when using S3boto backend +* Fixed key handling in S3boto and Google Storage backends +* Account for lack of multipart upload in Google Storage backend +* Fixed seek() issue when using AWS_IS_GZIPPED by darkness51 with pull-request `#50`_ +* Improvements to S3BotoStorage and GSBotoStorage + +.. _#50: https://bitbucket.org/david/django-storages/pull-request/50/ + +1.1.6 (2013-01-06) +****************** + +* Merged many changes from Jannis Leidel (mostly regarding gzipping) +* Fixed tests by Ian Lewis +* Added support for Google Cloud Storage backend by Jannis Leidel +* Updated license file by Dan Loewenherz, fixes `#133`_ with pull-request `#44`_ +* Set Content-Type header for use in upload_part_from_file by Gerardo Curiel +* Pass the rewind parameter to Boto's set_contents_from_file method by Jannis Leidel with pull-request `#45`_ +* Fix for FTPStorageFile close() method by Mathieu Comandon with pull-request `#43`_ +* Minor refactoring by Oktay Sancak with pull-request `#48`_ +* Ungzip on download based on Content-Encoding by Gavin Wahl with pull-request `#46`_ +* Add support for S3 server-side encryption by Tobias McNulty with pull-request `#17`_ +* Add an optional setting to the boto storage to produce protocol-relative URLs, fixes `#105`_ + +.. _#133: https://bitbucket.org/david/django-storages/issue/133/license-file-refers-to-incorrect-project +.. _#44: https://bitbucket.org/david/django-storages/pull-request/44/ +.. _#45: https://bitbucket.org/david/django-storages/pull-request/45/ +.. _#43: https://bitbucket.org/david/django-storages/pull-request/43/ +.. _#48: https://bitbucket.org/david/django-storages/pull-request/48/ +.. _#46: https://bitbucket.org/david/django-storages/pull-request/46/ +.. _#17: https://bitbucket.org/david/django-storages/pull-request/17/ +.. _#105: https://bitbucket.org/david/django-storages/issue/105/add-option-to-produce-protocol-relative + + +1.1.5 (2012-07-18) +****************** + +* Merged pull request `#36`_ from freakboy3742 Keith-Magee, improvements to Apache Libcloud backend and docs +* Merged pull request `#35`_ from atodorov, allows more granular S3 access settings +* Add support for SSL in Rackspace Cloudfiles backend +* Fixed the listdir() method in s3boto backend, fixes `#57`_ +* Added base url tests for safe_join in s3boto backend +* Merged pull request `#20`_ from alanjds, fixed SuspiciousOperation warning if AWS_LOCATION ends with '/' +* Added FILE_BUFFER_SIZE setting to s3boto backend +* Merged pull request `#30`_ from pendletongp, resolves `#108`_, `#109`_ and `#110`_ +* Updated the modified_time() method so that it doesn't require dateutil. fixes `#111`_ +* Merged pull request `#16`_ from chamal, adds Apache Libcloud backend +* When preloading the S3 metadata make sure we reset the files key during saving to prevent stale metadata +* Merged pull request `#24`_ from tobias.mcnulty, fixes bug where s3boto backend returns modified_time in wrong time zone +* Fixed HashPathStorage.location to no longer use settings.MEDIA_ROOT +* Remove download_url from setup file so PyPI dist is used + +.. _#36: https://bitbucket.org/david/django-storages/pull-request/36/ +.. _#35: https://bitbucket.org/david/django-storages/pull-request/35/ +.. _#57: https://bitbucket.org/david/django-storages/issue/57 +.. _#20: https://bitbucket.org/david/django-storages/pull-request/20/ +.. _#30: https://bitbucket.org/david/django-storages/pull-request/30/ +.. _#108: https://bitbucket.org/david/django-storages/issue/108 +.. _#109: https://bitbucket.org/david/django-storages/issue/109 +.. _#110: https://bitbucket.org/david/django-storages/issue/110 +.. _#111: https://bitbucket.org/david/django-storages/issue/111 +.. _#16: https://bitbucket.org/david/django-storages/pull-request/16/ +.. _#24: https://bitbucket.org/david/django-storages/pull-request/24/ + +1.1.4 (2012-01-06) +****************** + +* Added PendingDeprecationWarning for mosso backend +* Merged pull request `#13`_ from marcoala, adds ``SFTP_KNOWN_HOST_FILE`` setting to SFTP storage backend +* Merged pull request `#12`_ from ryankask, fixes HashPathStorage tests that delete remote media +* Merged pull request `#10`_ from key, adds support for django-mongodb-engine 0.4.0 or later, fixes GridFS file deletion bug +* Fixed S3BotoStorage performance problem calling modified_time() +* Added deprecation warning for s3 backend, refs `#40`_ +* Fixed CLOUDFILES_CONNECTION_KWARGS import error, fixes `#78`_ +* Switched to sphinx documentation, set official docs up on http://django-storages.rtfd.org/ +* HashPathStorage uses self.exists now, fixes `#83`_ + +.. _#13: https://bitbucket.org/david/django-storages/pull-request/13/a-version-of-sftp-storage-that-allows-you +.. _#12: https://bitbucket.org/david/django-storages/pull-request/12/hashpathstorage-tests-deleted-my-projects +.. _#10: https://bitbucket.org/david/django-storages/pull-request/10/support-django-mongodb-engine-040 +.. _#40: https://bitbucket.org/david/django-storages/issue/40/deprecate-s3py-backend +.. _#78: https://bitbucket.org/david/django-storages/issue/78/import-error +.. _#83: https://bitbucket.org/david/django-storages/issue/6/symlinkorcopystorage-new-custom-storage + +1.1.3 (2011-08-15) +****************** + +* Created this lovely change log +* Fixed `#89`_: broken StringIO import in CloudFiles backend +* Merged `pull request #5`_: HashPathStorage path bug + +.. _#89: https://bitbucket.org/david/django-storages/issue/89/112-broke-the-mosso-backend +.. _pull request #5: https://bitbucket.org/david/django-storages/pull-request/5/fixed-path-bug-and-added-testcase-for + @@ -0,0 +1,27 @@ +Copyright (c) 2008-2013, see AUTHORS file. +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: + + 1. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + + 3. Neither the name of django-storages nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON +ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. @@ -0,0 +1,9 @@ +=============== +Django storages +=============== + +Install +======= + + +See http://django-storages.readthedocs.org/ @@ -0,0 +1,618 @@ +#!/usr/bin/env python + +# This software code is made available "AS IS" without warranties of any +# kind. You may copy, display, modify and redistribute the software +# code either by itself or as incorporated into your code; provided that +# you do not remove any proprietary notices. Your use of this software +# code is at your own risk and you waive any claim against Amazon +# Digital Services, Inc. or its affiliates with respect to your use of +# this software code. (c) 2006-2007 Amazon Digital Services, Inc. or its +# affiliates. + +import base64 +import hmac +import httplib +try: + from hashlib import sha1 as sha +except ImportError: + import sha +import time +import urllib +import urlparse +import xml.sax + +DEFAULT_HOST = 's3.amazonaws.com' +PORTS_BY_SECURITY = { True: 443, False: 80 } +METADATA_PREFIX = 'x-amz-meta-' +AMAZON_HEADER_PREFIX = 'x-amz-' + +# generates the aws canonical string for the given parameters +def canonical_string(method, bucket="", key="", query_args={}, headers={}, expires=None): + interesting_headers = {} + for header_key in headers: + lk = header_key.lower() + if lk in ['content-md5', 'content-type', 'date'] or lk.startswith(AMAZON_HEADER_PREFIX): + interesting_headers[lk] = headers[header_key].strip() + + # these keys get empty strings if they don't exist + if not interesting_headers.has_key('content-type'): + interesting_headers['content-type'] = '' + if not interesting_headers.has_key('content-md5'): + interesting_headers['content-md5'] = '' + + # just in case someone used this. it's not necessary in this lib. + if interesting_headers.has_key('x-amz-date'): + interesting_headers['date'] = '' + + # if you're using expires for query string auth, then it trumps date + # (and x-amz-date) + if expires: + interesting_headers['date'] = str(expires) + + sorted_header_keys = interesting_headers.keys() + sorted_header_keys.sort() + + buf = "%s\n" % method + for header_key in sorted_header_keys: + if header_key.startswith(AMAZON_HEADER_PREFIX): + buf += "%s:%s\n" % (header_key, interesting_headers[header_key]) + else: + buf += "%s\n" % interesting_headers[header_key] + + # append the bucket if it exists + if bucket != "": + buf += "/%s" % bucket + + # add the key. even if it doesn't exist, add the slash + buf += "/%s" % urllib.quote_plus(key) + + # handle special query string arguments + + if query_args.has_key("acl"): + buf += "?acl" + elif query_args.has_key("torrent"): + buf += "?torrent" + elif query_args.has_key("logging"): + buf += "?logging" + elif query_args.has_key("location"): + buf += "?location" + + return buf + +# computes the base64'ed hmac-sha hash of the canonical string and the secret +# access key, optionally urlencoding the result +def encode(aws_secret_access_key, str, urlencode=False): + b64_hmac = base64.encodestring(hmac.new(aws_secret_access_key, str, sha).digest()).strip() + if urlencode: + return urllib.quote_plus(b64_hmac) + else: + return b64_hmac + +def merge_meta(headers, metadata): + final_headers = headers.copy() + for k in metadata.keys(): + final_headers[METADATA_PREFIX + k] = metadata[k] + + return final_headers + +# builds the query arg string +def query_args_hash_to_string(query_args): + query_string = "" + pairs = [] + for k, v in query_args.items(): + piece = k + if v != None: + piece += "=%s" % urllib.quote_plus(str(v)) + pairs.append(piece) + + return '&'.join(pairs) + + +class CallingFormat: + PATH = 1 + SUBDOMAIN = 2 + VANITY = 3 + + def build_url_base(protocol, server, port, bucket, calling_format): + url_base = '%s://' % protocol + + if bucket == '': + url_base += server + elif calling_format == CallingFormat.SUBDOMAIN: + url_base += "%s.%s" % (bucket, server) + elif calling_format == CallingFormat.VANITY: + url_base += bucket + else: + url_base += server + + url_base += ":%s" % port + + if (bucket != '') and (calling_format == CallingFormat.PATH): + url_base += "/%s" % bucket + + return url_base + + build_url_base = staticmethod(build_url_base) + + + +class Location: + DEFAULT = None + EU = 'EU' + + + +class AWSAuthConnection: + def __init__(self, aws_access_key_id, aws_secret_access_key, is_secure=True, + server=DEFAULT_HOST, port=None, calling_format=CallingFormat.SUBDOMAIN): + + if not port: + port = PORTS_BY_SECURITY[is_secure] + + self.aws_access_key_id = aws_access_key_id + self.aws_secret_access_key = aws_secret_access_key + self.is_secure = is_secure + self.server = server + self.port = port + self.calling_format = calling_format + + def create_bucket(self, bucket, headers={}): + return Response(self._make_request('PUT', bucket, '', {}, headers)) + + def create_located_bucket(self, bucket, location=Location.DEFAULT, headers={}): + if location == Location.DEFAULT: + body = "" + else: + body = "<CreateBucketConstraint><LocationConstraint>" + \ + location + \ + "</LocationConstraint></CreateBucketConstraint>" + return Response(self._make_request('PUT', bucket, '', {}, headers, body)) + + def check_bucket_exists(self, bucket): + return self._make_request('HEAD', bucket, '', {}, {}) + + def list_bucket(self, bucket, options={}, headers={}): + return ListBucketResponse(self._make_request('GET', bucket, '', options, headers)) + + def delete_bucket(self, bucket, headers={}): + return Response(self._make_request('DELETE', bucket, '', {}, headers)) + + def put(self, bucket, key, object, headers={}): + if not isinstance(object, S3Object): + object = S3Object(object) + + return Response( + self._make_request( + 'PUT', + bucket, + key, + {}, + headers, + object.data, + object.metadata)) + + def get(self, bucket, key, headers={}): + return GetResponse( + self._make_request('GET', bucket, key, {}, headers)) + + def delete(self, bucket, key, headers={}): + return Response( + self._make_request('DELETE', bucket, key, {}, headers)) + + def get_bucket_logging(self, bucket, headers={}): + return GetResponse(self._make_request('GET', bucket, '', { 'logging': None }, headers)) + + def put_bucket_logging(self, bucket, logging_xml_doc, headers={}): + return Response(self._make_request('PUT', bucket, '', { 'logging': None }, headers, logging_xml_doc)) + + def get_bucket_acl(self, bucket, headers={}): + return self.get_acl(bucket, '', headers) + + def get_acl(self, bucket, key, headers={}): + return GetResponse( + self._make_request('GET', bucket, key, { 'acl': None }, headers)) + + def put_bucket_acl(self, bucket, acl_xml_document, headers={}): + return self.put_acl(bucket, '', acl_xml_document, headers) + + def put_acl(self, bucket, key, acl_xml_document, headers={}): + return Response( + self._make_request( + 'PUT', + bucket, + key, + { 'acl': None }, + headers, + acl_xml_document)) + + def list_all_my_buckets(self, headers={}): + return ListAllMyBucketsResponse(self._make_request('GET', '', '', {}, headers)) + + def get_bucket_location(self, bucket): + return LocationResponse(self._make_request('GET', bucket, '', {'location' : None})) + + # end public methods + + def _make_request(self, method, bucket='', key='', query_args={}, headers={}, data='', metadata={}): + + server = '' + if bucket == '': + server = self.server + elif self.calling_format == CallingFormat.SUBDOMAIN: + server = "%s.%s" % (bucket, self.server) + elif self.calling_format == CallingFormat.VANITY: + server = bucket + else: + server = self.server + + path = '' + + if (bucket != '') and (self.calling_format == CallingFormat.PATH): + path += "/%s" % bucket + + # add the slash after the bucket regardless + # the key will be appended if it is non-empty + path += "/%s" % urllib.quote_plus(key) + + + # build the path_argument string + # add the ? in all cases since + # signature and credentials follow path args + if len(query_args): + path += "?" + query_args_hash_to_string(query_args) + + is_secure = self.is_secure + host = "%s:%d" % (server, self.port) + while True: + if (is_secure): + connection = httplib.HTTPSConnection(host) + else: + connection = httplib.HTTPConnection(host) + + final_headers = merge_meta(headers, metadata); + # add auth header + self._add_aws_auth_header(final_headers, method, bucket, key, query_args) + + connection.request(method, path, data, final_headers) + resp = connection.getresponse() + if resp.status < 300 or resp.status >= 400: + return resp + # handle redirect + location = resp.getheader('location') + if not location: + return resp + # (close connection) + resp.read() + scheme, host, path, params, query, fragment \ + = urlparse.urlparse(location) + if scheme == "http": is_secure = True + elif scheme == "https": is_secure = False + else: raise IOError("Not http/https: " + location) + if query: path += "?" + query + # retry with redirect + + def _add_aws_auth_header(self, headers, method, bucket, key, query_args): + if not headers.has_key('Date'): + headers['Date'] = time.strftime("%a, %d %b %Y %X GMT", time.gmtime()) + + c_string = canonical_string(method, bucket, key, query_args, headers) + headers['Authorization'] = \ + "AWS %s:%s" % (self.aws_access_key_id, encode(self.aws_secret_access_key, c_string)) + + +class QueryStringAuthGenerator: + # by default, expire in 1 minute + DEFAULT_EXPIRES_IN = 60 + + def __init__(self, aws_access_key_id, aws_secret_access_key, is_secure=True, + server=DEFAULT_HOST, port=None, calling_format=CallingFormat.SUBDOMAIN): + + if not port: + port = PORTS_BY_SECURITY[is_secure] + + self.aws_access_key_id = aws_access_key_id + self.aws_secret_access_key = aws_secret_access_key + if (is_secure): + self.protocol = 'https' + else: + self.protocol = 'http' + + self.is_secure = is_secure + self.server = server + self.port = port + self.calling_format = calling_format + self.__expires_in = QueryStringAuthGenerator.DEFAULT_EXPIRES_IN + self.__expires = None + + # for backwards compatibility with older versions + self.server_name = "%s:%s" % (self.server, self.port) + + def set_expires_in(self, expires_in): + self.__expires_in = expires_in + self.__expires = None + + def set_expires(self, expires): + self.__expires = expires + self.__expires_in = None + + def create_bucket(self, bucket, headers={}): + return self.generate_url('PUT', bucket, '', {}, headers) + + def list_bucket(self, bucket, options={}, headers={}): + return self.generate_url('GET', bucket, '', options, headers) + + def delete_bucket(self, bucket, headers={}): + return self.generate_url('DELETE', bucket, '', {}, headers) + + def put(self, bucket, key, object, headers={}): + if not isinstance(object, S3Object): + object = S3Object(object) + + return self.generate_url( + 'PUT', + bucket, + key, + {}, + merge_meta(headers, object.metadata)) + + def get(self, bucket, key, headers={}): + return self.generate_url('GET', bucket, key, {}, headers) + + def delete(self, bucket, key, headers={}): + return self.generate_url('DELETE', bucket, key, {}, headers) + + def get_bucket_logging(self, bucket, headers={}): + return self.generate_url('GET', bucket, '', { 'logging': None }, headers) + + def put_bucket_logging(self, bucket, logging_xml_doc, headers={}): + return self.generate_url('PUT', bucket, '', { 'logging': None }, headers) + + def get_bucket_acl(self, bucket, headers={}): + return self.get_acl(bucket, '', headers) + + def get_acl(self, bucket, key='', headers={}): + return self.generate_url('GET', bucket, key, { 'acl': None }, headers) + + def put_bucket_acl(self, bucket, acl_xml_document, headers={}): + return self.put_acl(bucket, '', acl_xml_document, headers) + + # don't really care what the doc is here. + def put_acl(self, bucket, key, acl_xml_document, headers={}): + return self.generate_url('PUT', bucket, key, { 'acl': None }, headers) + + def list_all_my_buckets(self, headers={}): + return self.generate_url('GET', '', '', {}, headers) + + def make_bare_url(self, bucket, key=''): + full_url = self.generate_url(self, bucket, key) + return full_url[:full_url.index('?')] + + def generate_url(self, method, bucket='', key='', query_args={}, headers={}): + expires = 0 + if self.__expires_in != None: + expires = int(time.time() + self.__expires_in) + elif self.__expires != None: + expires = int(self.__expires) + else: + raise "Invalid expires state" + + canonical_str = canonical_string(method, bucket, key, query_args, headers, expires) + encoded_canonical = encode(self.aws_secret_access_key, canonical_str) + + url = CallingFormat.build_url_base(self.protocol, self.server, self.port, bucket, self.calling_format) + + url += "/%s" % urllib.quote_plus(key) + + query_args['Signature'] = encoded_canonical + query_args['Expires'] = expires + query_args['AWSAccessKeyId'] = self.aws_access_key_id + + url += "?%s" % query_args_hash_to_string(query_args) + + return url + + +class S3Object: + def __init__(self, data, metadata={}): + self.data = data + self.metadata = metadata + +class Owner: + def __init__(self, id='', display_name=''): + self.id = id + self.display_name = display_name + +class ListEntry: + def __init__(self, key='', last_modified=None, etag='', size=0, storage_class='', owner=None): + self.key = key + self.last_modified = last_modified + self.etag = etag + self.size = size + self.storage_class = storage_class + self.owner = owner + +class CommonPrefixEntry: + def __init(self, prefix=''): + self.prefix = prefix + +class Bucket: + def __init__(self, name='', creation_date=''): + self.name = name + self.creation_date = creation_date + +class Response: + def __init__(self, http_response): + self.http_response = http_response + # you have to do this read, even if you don't expect a body. + # otherwise, the next request fails. + self.body = http_response.read() + if http_response.status >= 300 and self.body: + self.message = self.body + else: + self.message = "%03d %s" % (http_response.status, http_response.reason) + + + +class ListBucketResponse(Response): + def __init__(self, http_response): + Response.__init__(self, http_response) + if http_response.status < 300: + handler = ListBucketHandler() + xml.sax.parseString(self.body, handler) + self.entries = handler.entries + self.common_prefixes = handler.common_prefixes + self.name = handler.name + self.marker = handler.marker + self.prefix = handler.prefix + self.is_truncated = handler.is_truncated + self.delimiter = handler.delimiter + self.max_keys = handler.max_keys + self.next_marker = handler.next_marker + else: + self.entries = [] + +class ListAllMyBucketsResponse(Response): + def __init__(self, http_response): + Response.__init__(self, http_response) + if http_response.status < 300: + handler = ListAllMyBucketsHandler() + xml.sax.parseString(self.body, handler) + self.entries = handler.entries + else: + self.entries = [] + +class GetResponse(Response): + def __init__(self, http_response): + Response.__init__(self, http_response) + response_headers = http_response.msg # older pythons don't have getheaders + metadata = self.get_aws_metadata(response_headers) + self.object = S3Object(self.body, metadata) + + def get_aws_metadata(self, headers): + metadata = {} + for hkey in headers.keys(): + if hkey.lower().startswith(METADATA_PREFIX): + metadata[hkey[len(METADATA_PREFIX):]] = headers[hkey] + del headers[hkey] + + return metadata + +class LocationResponse(Response): + def __init__(self, http_response): + Response.__init__(self, http_response) + if http_response.status < 300: + handler = LocationHandler() + xml.sax.parseString(self.body, handler) + self.location = handler.location + +class ListBucketHandler(xml.sax.ContentHandler): + def __init__(self): + self.entries = [] + self.curr_entry = None + self.curr_text = '' + self.common_prefixes = [] + self.curr_common_prefix = None + self.name = '' + self.marker = '' + self.prefix = '' + self.is_truncated = False + self.delimiter = '' + self.max_keys = 0 + self.next_marker = '' + self.is_echoed_prefix_set = False + + def startElement(self, name, attrs): + if name == 'Contents': + self.curr_entry = ListEntry() + elif name == 'Owner': + self.curr_entry.owner = Owner() + elif name == 'CommonPrefixes': + self.curr_common_prefix = CommonPrefixEntry() + + + def endElement(self, name): + if name == 'Contents': + self.entries.append(self.curr_entry) + elif name == 'CommonPrefixes': + self.common_prefixes.append(self.curr_common_prefix) + elif name == 'Key': + self.curr_entry.key = self.curr_text + elif name == 'LastModified': + self.curr_entry.last_modified = self.curr_text + elif name == 'ETag': + self.curr_entry.etag = self.curr_text + elif name == 'Size': + self.curr_entry.size = int(self.curr_text) + elif name == 'ID': + self.curr_entry.owner.id = self.curr_text + elif name == 'DisplayName': + self.curr_entry.owner.display_name = self.curr_text + elif name == 'StorageClass': + self.curr_entry.storage_class = self.curr_text + elif name == 'Name': + self.name = self.curr_text + elif name == 'Prefix' and self.is_echoed_prefix_set: + self.curr_common_prefix.prefix = self.curr_text + elif name == 'Prefix': + self.prefix = self.curr_text + self.is_echoed_prefix_set = True + elif name == 'Marker': + self.marker = self.curr_text + elif name == 'IsTruncated': + self.is_truncated = self.curr_text == 'true' + elif name == 'Delimiter': + self.delimiter = self.curr_text + elif name == 'MaxKeys': + self.max_keys = int(self.curr_text) + elif name == 'NextMarker': + self.next_marker = self.curr_text + + self.curr_text = '' + + def characters(self, content): + self.curr_text += content + + +class ListAllMyBucketsHandler(xml.sax.ContentHandler): + def __init__(self): + self.entries = [] + self.curr_entry = None + self.curr_text = '' + + def startElement(self, name, attrs): + if name == 'Bucket': + self.curr_entry = Bucket() + + def endElement(self, name): + if name == 'Name': + self.curr_entry.name = self.curr_text + elif name == 'CreationDate': + self.curr_entry.creation_date = self.curr_text + elif name == 'Bucket': + self.entries.append(self.curr_entry) + + def characters(self, content): + self.curr_text = content + + +class LocationHandler(xml.sax.ContentHandler): + def __init__(self): + self.location = None + self.state = 'init' + + def startElement(self, name, attrs): + if self.state == 'init': + if name == 'LocationConstraint': + self.state = 'tag_location' + self.location = '' + else: self.state = 'bad' + else: self.state = 'bad' + + def endElement(self, name): + if self.state == 'tag_location' and name == 'LocationConstraint': + self.state = 'done' + else: self.state = 'bad' + + def characters(self, content): + if self.state == 'tag_location': + self.location += content diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000..18f6e2d --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,130 @@ +# Makefile for Sphinx documentation +# + +# You can set these variables from the command line. +SPHINXOPTS = +SPHINXBUILD = sphinx-build +PAPER = +BUILDDIR = _build + +# Internal variables. +PAPEROPT_a4 = -D latex_paper_size=a4 +PAPEROPT_letter = -D latex_paper_size=letter +ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . + +.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest + +help: + @echo "Please use \`make <target>' where <target> is one of" + @echo " html to make standalone HTML files" + @echo " dirhtml to make HTML files named index.html in directories" + @echo " singlehtml to make a single large HTML file" + @echo " pickle to make pickle files" + @echo " json to make JSON files" + @echo " htmlhelp to make HTML files and a HTML help project" + @echo " qthelp to make HTML files and a qthelp project" + @echo " devhelp to make HTML files and a Devhelp project" + @echo " epub to make an epub" + @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" + @echo " latexpdf to make LaTeX files and run them through pdflatex" + @echo " text to make text files" + @echo " man to make manual pages" + @echo " changes to make an overview of all changed/added/deprecated items" + @echo " linkcheck to check all external links for integrity" + @echo " doctest to run all doctests embedded in the documentation (if enabled)" + +clean: + -rm -rf $(BUILDDIR)/* + +html: + $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html + @echo + @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." + +dirhtml: + $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml + @echo + @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." + +singlehtml: + $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml + @echo + @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." + +pickle: + $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle + @echo + @echo "Build finished; now you can process the pickle files." + +json: + $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json + @echo + @echo "Build finished; now you can process the JSON files." + +htmlhelp: + $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp + @echo + @echo "Build finished; now you can run HTML Help Workshop with the" \ + ".hhp project file in $(BUILDDIR)/htmlhelp." + +qthelp: + $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp + @echo + @echo "Build finished; now you can run "qcollectiongenerator" with the" \ + ".qhcp project file in $(BUILDDIR)/qthelp, like this:" + @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/django-storages.qhcp" + @echo "To view the help file:" + @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/django-storages.qhc" + +devhelp: + $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp + @echo + @echo "Build finished." + @echo "To view the help file:" + @echo "# mkdir -p $$HOME/.local/share/devhelp/django-storages" + @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/django-storages" + @echo "# devhelp" + +epub: + $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub + @echo + @echo "Build finished. The epub file is in $(BUILDDIR)/epub." + +latex: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo + @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." + @echo "Run \`make' in that directory to run these through (pdf)latex" \ + "(use \`make latexpdf' here to do that automatically)." + +latexpdf: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo "Running LaTeX files through pdflatex..." + make -C $(BUILDDIR)/latex all-pdf + @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." + +text: + $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text + @echo + @echo "Build finished. The text files are in $(BUILDDIR)/text." + +man: + $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man + @echo + @echo "Build finished. The manual pages are in $(BUILDDIR)/man." + +changes: + $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes + @echo + @echo "The overview file is in $(BUILDDIR)/changes." + +linkcheck: + $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck + @echo + @echo "Link check complete; look for any errors in the above output " \ + "or in $(BUILDDIR)/linkcheck/output.txt." + +doctest: + $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest + @echo "Testing of doctests in the sources finished, look at the " \ + "results in $(BUILDDIR)/doctest/output.txt." diff --git a/docs/backends/amazon-S3.rst b/docs/backends/amazon-S3.rst new file mode 100644 index 0000000..3773c02 --- /dev/null +++ b/docs/backends/amazon-S3.rst @@ -0,0 +1,202 @@ +Amazon S3 +========= + +Usage +***** + +There are two backend APIs for interacting with S3. The first is the s3 backend (in storages/backends/s3.py) which is simple and based on the Amazon S3 Python library. The second is the s3boto backend (in storages/backends/s3boto.py) which is well-maintained by the community and is generally more robust (including connection pooling, etc...). s3boto requires the python-boto library. + +Settings +-------- + +``DEFAULT_FILE_STORAGE`` + +This setting sets the path to the S3 storage class, the first part correspond to the filepath and the second the name of the class, if you've got example.com in your PYTHONPATH and store your storage file in example.com/libs/storages/S3Storage.py, the resulting setting will be:: + + DEFAULT_FILE_STORAGE = 'libs.storages.S3Storage.S3Storage' + +or if you installed using setup.py:: + + DEFAULT_FILE_STORAGE = 'storages.backends.s3.S3Storage' + +If you keep the same filename as in repository, it should always end with S3Storage.S3Storage. + +To use s3boto, this setting will be:: + + DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage' + +``AWS_ACCESS_KEY_ID`` + +Your Amazon Web Services access key, as a string. + +``AWS_SECRET_ACCESS_KEY`` + +Your Amazon Web Services secret access key, as a string. + +``AWS_STORAGE_BUCKET_NAME`` + +Your Amazon Web Services storage bucket name, as a string. + +``AWS_CALLING_FORMAT`` (Subdomain hardcoded in s3boto) + +The way you'd like to call the Amazon Web Services API, for instance if you prefer subdomains:: + + from S3 import CallingFormat + AWS_CALLING_FORMAT = CallingFormat.SUBDOMAIN + +``AWS_HEADERS`` (optional) + +If you'd like to set headers sent with each file of the storage:: + + # see http://developer.yahoo.com/performance/rules.html#expires + AWS_HEADERS = { + 'Expires': 'Thu, 15 Apr 2010 20:00:00 GMT', + 'Cache-Control': 'max-age=86400', + } + +To allow ``django-admin.py`` collectstatic to automatically put your static files in your bucket set the following in your settings.py:: + + STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage' + +Fields +------ + +Once you're done, default_storage will be the S3 storage:: + + >>> from django.core.files.storage import default_storage + >>> print default_storage.__class__ + <class 'S3Storage.S3Storage'> + +The above doesn't seem to be true for django 1.3+ instead look at:: + + >>> from django.core.files.storage import default_storage + >>> print default_storage.connection + S3Connection:s3.amazonaws.com + +This way, if you define a new FileField, it will use the S3 storage:: + + >>> from django.db import models + >>> class Resume(models.Model): + ... pdf = models.FileField(upload_to='pdfs') + ... photos = models.ImageField(upload_to='photos') + ... + >>> resume = Resume() + >>> print resume.pdf.storage + <S3Storage.S3Storage object at ...> + +Tests +***** + +Initialization:: + + >>> from django.core.files.storage import default_storage + >>> from django.core.files.base import ContentFile + >>> from django.core.cache import cache + >>> from models import MyStorage + +Storage +------- + +Standard file access options are available, and work as expected:: + + >>> default_storage.exists('storage_test') + False + >>> file = default_storage.open('storage_test', 'w') + >>> file.write('storage contents') + >>> file.close() + + >>> default_storage.exists('storage_test') + True + >>> file = default_storage.open('storage_test', 'r') + >>> file.read() + 'storage contents' + >>> file.close() + + >>> default_storage.delete('storage_test') + >>> default_storage.exists('storage_test') + False + +Model +----- + +An object without a file has limited functionality:: + + >>> obj1 = MyStorage() + >>> obj1.normal + <FieldFile: None> + >>> obj1.normal.size + Traceback (most recent call last): + ... + ValueError: The 'normal' attribute has no file associated with it. + +Saving a file enables full functionality:: + + >>> obj1.normal.save('django_test.txt', ContentFile('content')) + >>> obj1.normal + <FieldFile: tests/django_test.txt> + >>> obj1.normal.size + 7 + >>> obj1.normal.read() + 'content' + +Files can be read in a little at a time, if necessary:: + + >>> obj1.normal.open() + >>> obj1.normal.read(3) + 'con' + >>> obj1.normal.read() + 'tent' + >>> '-'.join(obj1.normal.chunks(chunk_size=2)) + 'co-nt-en-t' + +Save another file with the same name:: + + >>> obj2 = MyStorage() + >>> obj2.normal.save('django_test.txt', ContentFile('more content')) + >>> obj2.normal + <FieldFile: tests/django_test_.txt> + >>> obj2.normal.size + 12 + +Push the objects into the cache to make sure they pickle properly:: + + >>> cache.set('obj1', obj1) + >>> cache.set('obj2', obj2) + >>> cache.get('obj2').normal + <FieldFile: tests/django_test_.txt> + +Deleting an object deletes the file it uses, if there are no other objects still using that file:: + + >>> obj2.delete() + >>> obj2.normal.save('django_test.txt', ContentFile('more content')) + >>> obj2.normal + <FieldFile: tests/django_test_.txt> + +Default values allow an object to access a single file:: + + >>> obj3 = MyStorage.objects.create() + >>> obj3.default + <FieldFile: tests/default.txt> + >>> obj3.default.read() + 'default content' + +But it shouldn't be deleted, even if there are no more objects using it:: + + >>> obj3.delete() + >>> obj3 = MyStorage() + >>> obj3.default.read() + 'default content' + +Verify the fix for #5655, making sure the directory is only determined once:: + + >>> obj4 = MyStorage() + >>> obj4.random.save('random_file', ContentFile('random content')) + >>> obj4.random + <FieldFile: .../random_file> + +Clean up the temporary files:: + + >>> obj1.normal.delete() + >>> obj2.normal.delete() + >>> obj3.default.delete() + >>> obj4.random.delete() diff --git a/docs/backends/apache_libcloud.rst b/docs/backends/apache_libcloud.rst new file mode 100644 index 0000000..b53401b --- /dev/null +++ b/docs/backends/apache_libcloud.rst @@ -0,0 +1,174 @@ +Apache Libcloud +=============== + +`Apache Libcloud`_ is an API wrapper around a range of cloud storage providers. +It aims to provide a consistent API for dealing with cloud storage (and, more +broadly, the many other services provided by cloud providers, such as device +provisioning, load balancer configuration, and DNS configuration). + +As of v0.10.1, Libcloud supports the following cloud storage providers: + * `Amazon S3`_ + * `Google Cloud Storage`_ + * `Nimbus.io`_ + * `Ninefold Cloud Storage`_ + * `Rackspace CloudFiles`_ + +Libcloud can also be configured with relatively little effort to support any provider +using EMC Atmos storage, or the OpenStack API. + +.. _Apache Libcloud: http://libcloud.apache.org/ +.. _Amazon S3: http://aws.amazon.com/s3/ +.. _Google Cloud Storage: http://cloud.google.com/products/cloud-storage.html +.. _Rackspace CloudFiles: http://www.rackspace.com/cloud/cloud_hosting_products/files/ +.. _Ninefold Cloud Storage: http://ninefold.com/cloud-storage/ +.. _Nimbus.io: http://nimbus.io + +Settings +-------- + +``LIBCLOUD_PROVIDERS`` +~~~~~~~~~~~~~~~~~~~~~~ + +This setting is required to configure connections to cloud storage providers. +Each entry corresponds to a single 'bucket' of storage. You can have multiple +buckets for a single service provider (e.g., multiple S3 buckets), and you can +define buckets at multiple providers. For example, the following configuration +defines 3 providers: two buckets (``bucket-1`` and ``bucket-2``) on a US-based +Amazon S3 store, and a third bucket (``bucket-3``) on Google:: + + + LIBCLOUD_PROVIDERS = { + 'amazon_1': { + 'type': 'libcloud.storage.types.Provider.S3_US_STANDARD_HOST', + 'user': '<your username here>', + 'key': '<your key here>', + 'bucket': 'bucket-1', + }, + 'amazon_2': { + 'type': 'libcloud.storage.types.Provider.S3_US_STANDARD_HOST', + 'user': '<your username here>', + 'key': '<your key here>', + 'bucket': 'bucket-2', + }, + 'google': { + 'type': 'libcloud.storage.types.GOOGLE_STORAGE', + 'user': '<Your Google APIv1 username>', + 'key': '<Your Google APIv1 Key>', + 'bucket': 'bucket-3', + }, + } + +The values for the ``type``, ``user`` and ``key`` arguments will vary depending on +your storage provider: + + **Amazon S3**: + + **type**: ``libcloud.storage.types.Provider.S3_US_STANDARD_HOST``, + + **user**: Your AWS access key ID + + **key**: Your AWS secret access key + + If you want to use a availability zone other than the US default, you + can use one of ``S3_US_WEST_HOST``, ``S3_US_WEST_OREGON_HOST``, + ``S3_EU_WEST_HOST``, ``S3_AP_SOUTHEAST_HOST``, or + ``S3_AP_NORTHEAST_HOST`` instead of ``S3_US_STANDARD_HOST``. + + **Google Cloud Storage**: + + **type**: ``libcloud.storage.types.Provider.GOOGLE_STORAGE``, + + **user**: Your Google APIv1 username (20 characters) + + **key**: Your Google APIv1 key + + **Nimbus.io**: + + **type**: ``libcloud.storage.types.Provider.NIMBUS``, + + **user**: Your Nimbus.io user ID + + **key**: Your Nimbus.io access key + + **Ninefold Cloud Storage**: + + **type**: ``libcloud.storage.types.Provider.NINEFOLD``, + + **user**: Your Atmos Access Token + + **key**: Your Atmos Shared Secret + + **Rackspace Cloudfiles**: + + **type**: ``libcloud.storage.types.Provider.CLOUDFIULES_US`` or ``libcloud.storage.types.Provider.CLOUDFIULES_UK``, + + **user**: Your Rackspace user ID + + **key**: Your Rackspace access key + +You can specify any bucket name you want; however, the bucket must exist before you +can start using it. If you need to create the bucket, you can use the storage API. +For example, to create ``bucket-1`` from our previous example:: + + >>> from storages.backends.apache_libcloud import LibCloudStorage + >>> store = LibCloudStorage('amazon_1') + >>> store.driver.create_container('bucket-1') + + +``DEFAULT_LIBCLOUD_PROVIDER`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once you have defined your Libcloud providers, you have the option of +setting one provider as the default provider of Libcloud storage. This +is done setting ``DEFAULT_LIBCLOUD_PROVIDER`` to the key in +``LIBCLOUD_PROVIDER`` that you want to use as the default provider. +For example, if you want the ``amazon-1`` provider to be the default +provider, use:: + + DEFAULT_LIBCLOUD_PROVIDER = 'amazon-1' + +If ``DEFAULT_LIBCLOUD_PROVIDER`` isn't set, the Libcloud backend will assume +that the default storage backend is named ``default``. Therefore, you can +avoid settings DEFAULT_LIBCLOUD_PROVIDER by simply naming one of your +Libcloud providers ``default``:: + + LIBCLOUD_PROVIDERS = { + 'default': { + 'type': ... + }, + } + + +``DEFAULT_FILE_STORAGE`` +~~~~~~~~~~~~~~~~~~~~~~~~ + +If you want your Libcloud storage to be the default Django file store, you can +set:: + + DEFAULT_FILE_STORAGE = 'storages.backends.apache_libcloud.LibCloudStorage' + +Your default Libcloud provider will be used as the file store. + +Certifcate authorities +---------------------- + +Libcloud uses HTTPS connections, and in order for these HTTPS connections are +correctly signed, certificate authorities must be present. On some platforms +(most notably, OS X and Windows), the required certificates may not be available +by default. To test + + >>> from storages.backends.apache_libcloud import LibCloudStorage + >>> store = LibCloudStorage('amazon_1') + Traceback (most recent call last): + ... + ImproperlyConfigured: Unable to create libcloud driver type libcloud.storage.types.Provider.S3_US_STANDARD_HOST: No CA Certificates were found in CA_CERTS_PATH. + +If you get this error, you need to install a certificate authority. +`Download a certificate authority file`_, and then put the following two lines +into your settings.py:: + + import libcloud.security + libcloud.security.CA_CERTS_PATH.append("/path/to/your/cacerts.pem") + +.. _Download a certificate authority file: http://curl.haxx.se/ca/cacert.pem + diff --git a/docs/backends/azure.rst b/docs/backends/azure.rst new file mode 100644 index 0000000..6433b17 --- /dev/null +++ b/docs/backends/azure.rst @@ -0,0 +1,32 @@ +Azure Storage +=========== + +A custom storage system for Django using Windows Azure Storage backend. + + +Settings +******* + +``DEFAULT_FILE_STORAGE`` + +This setting sets the path to the Azure storage class:: + + DEFAULT_FILE_STORAGE = 'storages.backends.azure_storage.AzureStorage' + + +``AZURE_ACCOUNT_NAME`` + +This setting is the Windows Azure Storage Account name, which in many cases is also the first part of the url for instance: http://azure_account_name.blob.core.windows.net/ would mean:: + + AZURE_ACCOUNT_NAME = "azure_account_name" + +``AZURE_ACCOUNT_KEY`` + +This is the private key that gives your Django app access to your Windows Azure Account. + +``AZURE_CONTAINER`` + +This is where the files uploaded through your Django app will be uploaded. +The container must be already created as the storage system will not attempt to create it. + + diff --git a/docs/backends/couchdb.rst b/docs/backends/couchdb.rst new file mode 100644 index 0000000..f93761b --- /dev/null +++ b/docs/backends/couchdb.rst @@ -0,0 +1,5 @@ +CouchDB +======= + +A custom storage system for Django with CouchDB backend. + diff --git a/docs/backends/database.rst b/docs/backends/database.rst new file mode 100644 index 0000000..bc86e1c --- /dev/null +++ b/docs/backends/database.rst @@ -0,0 +1,59 @@ +Database +======== + +Class DatabaseStorage can be used with either FileField or ImageField. It can be used to map filenames to database blobs: so you have to use it with a special additional table created manually. The table should contain a pk-column for filenames (better to use the same type that FileField uses: nvarchar(100)), blob field (image type for example) and size field (bigint). You can't just create blob column in the same table, where you defined FileField, since there is no way to find required row in the save() method. Also size field is required to obtain better perfomance (see size() method). + +So you can use it with different FileFields and even with different "upload_to" variables used. Thus it implements a kind of root filesystem, where you can define dirs using "upload_to" with FileField and store any files in these dirs. + +It uses either settings.DB_FILES_URL or constructor param 'base_url' (see __init__()) to create urls to files. Base url should be mapped to view that provides access to files. To store files in the same table, where FileField is defined you have to define your own field and provide extra argument (e.g. pk) to save(). + +Raw sql is used for all operations. In constructor or in DB_FILES of settings.py () you should specify a dictionary with db_table, fname_column, blob_column, size_column and 'base_url'. For example I just put to the settings.py the following line:: + + DB_FILES = { + 'db_table': 'FILES', + 'fname_column': 'FILE_NAME', + 'blob_column': 'BLOB', + 'size_column': 'SIZE', + 'base_url': 'http://localhost/dbfiles/' + } + +And use it with ImageField as following:: + + player_photo = models.ImageField(upload_to="player_photos", storage=DatabaseStorage() ) + +DatabaseStorage class uses your settings.py file to perform custom connection to your database. + +The reason to use custom connection: http://code.djangoproject.com/ticket/5135 Connection string looks like:: + + cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=localhost;DATABASE=testdb;UID=me;PWD=pass') + +It's based on pyodbc module, so can be used with any database supported by pyodbc. I've tested it with MS Sql Express 2005. + +Note: It returns special path, which should be mapped to special view, which returns requested file:: + + def image_view(request, filename): + import os + from django.http import HttpResponse + from django.conf import settings + from django.utils._os import safe_join + from filestorage import DatabaseStorage + from django.core.exceptions import ObjectDoesNotExist + + storage = DatabaseStorage() + + try: + image_file = storage.open(filename, 'rb') + file_content = image_file.read() + except: + filename = 'no_image.gif' + path = safe_join(os.path.abspath(settings.MEDIA_ROOT), filename) + if not os.path.exists(path): + raise ObjectDoesNotExist + no_image = open(path, 'rb') + file_content = no_image.read() + + response = HttpResponse(file_content, mimetype="image/jpeg") + response['Content-Disposition'] = 'inline; filename=%s'%filename + return response + +Note: If filename exist, blob will be overwritten, to change this remove get_available_name(self, name), so Storage.get_available_name(self, name) will be used to generate new filename. diff --git a/docs/backends/ftp.rst b/docs/backends/ftp.rst new file mode 100644 index 0000000..6b894f0 --- /dev/null +++ b/docs/backends/ftp.rst @@ -0,0 +1,7 @@ +FTP +=== + +.. warning:: This FTP storage is not prepared to work with large files, because it uses memory for temporary data storage. It also does not close FTP connection automatically (but open it lazy and try to reestablish when disconnected). + +This implementation was done preliminary for upload files in admin to remote FTP location and read them back on site by HTTP. It was tested mostly in this configuration, so read/write using FTPStorageFile class may break. + diff --git a/docs/backends/image.rst b/docs/backends/image.rst new file mode 100644 index 0000000..b03e1d4 --- /dev/null +++ b/docs/backends/image.rst @@ -0,0 +1,5 @@ +Image +===== + +A custom FileSystemStorage made for normalizing extensions. It lets PIL look at the file to determine the format and append an always lower-case extension based on the results. + diff --git a/docs/backends/mogilefs.rst b/docs/backends/mogilefs.rst new file mode 100644 index 0000000..a330fdd --- /dev/null +++ b/docs/backends/mogilefs.rst @@ -0,0 +1,55 @@ +MogileFS +======== + +This storage allows you to use MogileFS, it comes from this blog post. + +The MogileFS storage backend is fairly simple: it uses URLs (or, rather, parts of URLs) as keys into the mogile database. When the user requests a file stored by mogile (say, an avatar), the URL gets passed to a view which, using a client to the mogile tracker, retrieves the "correct" path (the path that points to the actual file data). The view will then either return the path(s) to perlbal to reproxy, or, if you're not using perlbal to reproxy (which you should), it serves the data of the file directly from django. + +* ``MOGILEFS_DOMAIN``: The mogile domain that files should read from/written to, e.g "production" +* ``MOGILEFS_TRACKERS``: A list of trackers to connect to, e.g. ["foo.sample.com:7001", "bar.sample.com:7001"] +* ``MOGILEFS_MEDIA_URL`` (optional): The prefix for URLs that point to mogile files. This is used in a similar way to ``MEDIA_URL``, e.g. "/mogilefs/" +* ``SERVE_WITH_PERLBAL``: Boolean that, when True, will pass the paths back in the response in the ``X-REPROXY-URL`` header. If False, django will serve all mogile media files itself (bad idea for production, but useful if you're testing on a setup that doesn't have perlbal running) +* ``DEFAULT_FILE_STORAGE``: This is the class that's used for the backend. You'll want to set this to ``project.app.storages.MogileFSStorage`` (or wherever you've installed the backend) + +Getting files into mogile +************************* + +The great thing about file backends is that we just need to specify the backend in the model file and everything is taken care for us - all the default save() methods work correctly. + +For Fluther, we have two main media types we use mogile for: avatars and thumbnails. Mogile defines "classes" that dictate how each type of file is replicated - so you can make sure you have 3 copies of the original avatar but only 1 of the thumbnail. + +In order for classes to behave nicely with the backend framework, we've had to do a little tomfoolery. (This is something that may change in future versions of the filestorage framework). + +Here's what the models.py file looks like for the avatars:: + + from django.core.filestorage import storage + + # TODO: Find a better way to deal with classes. Maybe a generator? + class AvatarStorage(storage.__class__): + mogile_class = 'avatar' + + class ThumbnailStorage(storage.__class__): + mogile_class = 'thumb' + + class Avatar(models.Model): + user = models.ForeignKey(User, null=True, blank=True) + image = models.ImageField(storage=AvatarStorage()) + thumb = models.ImageField(storage=ThumbnailStorage()) + +Each of the custom storage classes defines a class attribute which gets passed to the mogile backend behind the scenes. If you don't want to worry about mogile classes, don't need to define a custom storage engine or specify it in the field - the default should work just fine. + +Serving files from mogile +************************* + +Now, all we need to do is plug in the view that serves up mogile data. + +Here's what we use:: + + urlpatterns += patterns(", + (r'^%s(?P<key>.*)' % settings.MOGILEFS_MEDIA_URL[1:], + 'MogileFSStorage.serve_mogilefs_file') + ) + +Any url beginning with the value of ``MOGILEFS_MEDIA_URL`` will get passed to our view. Since ``MOGILEFS_MEDIA_URL`` requires a leading slash (like ``MEDIA_URL``), we strip that off and pass the rest of the url over to the view. + +That's it! Happy mogiling! diff --git a/docs/backends/mongodb.rst b/docs/backends/mongodb.rst new file mode 100644 index 0000000..2ea997f --- /dev/null +++ b/docs/backends/mongodb.rst @@ -0,0 +1,21 @@ +MongoDB +======= + +A GridFS backend that works with django_mongodb_engine and the upcoming GSoC 2010 MongoDB backend which gets developed by Alex Gaynor. + +Usage (in settings.py):: + + DATABASES = { + 'default': { + 'ENGINE': 'django_mongodb_engine.mongodb', + 'NAME': 'test', + 'USER': '', + 'PASSWORD': '', + 'HOST': 'localhost', + 'PORT': 27017, + 'SUPPORTS_TRANSACTIONS': False, + } + } + + DEFAULT_FILE_STORAGE = 'storages.backends.mongodb.GridFSStorage' + GRIDFS_DATABASE = 'default' diff --git a/docs/backends/overwrite.rst b/docs/backends/overwrite.rst new file mode 100644 index 0000000..66aa875 --- /dev/null +++ b/docs/backends/overwrite.rst @@ -0,0 +1,5 @@ +Overwrite +========= + +This is a simple implementation overwrite of the FileSystemStorage. It removes the addition of an '_' to the filename if the file already exists in the storage system. I needed a model in the admin area to act exactly like a file system (overwriting the file if it already exists). + diff --git a/docs/backends/rackspace-cloudfiles.rst b/docs/backends/rackspace-cloudfiles.rst new file mode 100644 index 0000000..99c41d2 --- /dev/null +++ b/docs/backends/rackspace-cloudfiles.rst @@ -0,0 +1,37 @@ +Rackspace CloudFiles +==================== + +Requirements +************ + +Mosso's Cloud Files python module http://www.mosso.com/cloudfiles.jsp + +Usage +***** + +Add the following to your project's settings.py file:: + + CLOUDFILES_USERNAME = 'YourUsername' + CLOUDFILES_API_KEY = 'YourAPIKey' + CLOUDFILES_CONTAINER = 'ContainerName' + DEFAULT_FILE_STORAGE = 'backends.mosso.CloudFilesStorage' + + # Optional - use SSL + CLOUDFILES_SSL = True + +Optionally, you can implement the following custom upload_to in your models.py file. This will upload the file using the file name only to Cloud Files (e.g. 'myfile.jpg'). If you supply a string (e.g. upload_to='some/path'), your file name will include the path (e.g. 'some/path/myfile.jpg'):: + + from backends.mosso import cloudfiles_upload_to + + class SomeKlass(models.Model): + some_field = models.ImageField(upload_to=cloudfiles_upload_to) + +Alternatively, if you don't want to set the DEFAULT_FILE_STORAGE, you can do the following in your models:: + + from backends.mosso import CloudFilesStorage, cloudfiles_upload_to + + cloudfiles_storage = CloudFilesStorage() + + class SomeKlass(models.Model): + some_field = models.ImageField(storage=cloudfiles_storage, + upload_to=cloudfiles_upload_to) diff --git a/docs/backends/sftp.rst b/docs/backends/sftp.rst new file mode 100644 index 0000000..6f19f9f --- /dev/null +++ b/docs/backends/sftp.rst @@ -0,0 +1,5 @@ +SFTP +==== + +Take a look at the top of the backend's file for the documentation. + diff --git a/docs/backends/symlinkcopy.rst b/docs/backends/symlinkcopy.rst new file mode 100644 index 0000000..be4abe1 --- /dev/null +++ b/docs/backends/symlinkcopy.rst @@ -0,0 +1,6 @@ +Symlink or copy +=============== + +Stores symlinks to files instead of actual files whenever possible + +When a file that's being saved is currently stored in the symlink_within directory, then symlink the file. Otherwise, copy the file. diff --git a/docs/conf.py b/docs/conf.py new file mode 100644 index 0000000..c1052fa --- /dev/null +++ b/docs/conf.py @@ -0,0 +1,257 @@ +# -*- coding: utf-8 -*- +# +# django-storages documentation build configuration file, created by +# sphinx-quickstart on Sun Aug 28 13:44:45 2011. +# +# This file is execfile()d with the current directory set to its containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +import sys, os + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +sys.path.insert(0, os.path.abspath('..')) +import storages + +# -- General configuration ----------------------------------------------------- + +# If your documentation needs a minimal Sphinx version, state it here. +#needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be extensions +# coming with Sphinx (named 'sphinx.ext.*') or your custom ones. +extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix of source filenames. +source_suffix = '.rst' + +# The encoding of source files. +#source_encoding = 'utf-8-sig' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'django-storages' +copyright = u'2011-2013, David Larlet, et. al.' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = storages.__version__ +# The full version, including alpha/beta/rc tags. +release = storages.__version__ + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +#language = None + +# There are two options for replacing |today|: either, you set today to some +# non-false value, then it is used: +#today = '' +# Else, today_fmt is used as the format for a strftime call. +#today_fmt = '%B %d, %Y' + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +exclude_patterns = ['_build'] + +# The reST default role (used for this markup: `text`) to use for all documents. +#default_role = None + +# If true, '()' will be appended to :func: etc. cross-reference text. +#add_function_parentheses = True + +# If true, the current module name will be prepended to all description +# unit titles (such as .. function::). +#add_module_names = True + +# If true, sectionauthor and moduleauthor directives will be shown in the +# output. They are ignored by default. +#show_authors = False + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# A list of ignored prefixes for module index sorting. +#modindex_common_prefix = [] + + +# -- Options for HTML output --------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +html_theme = 'default' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +#html_theme_options = {} + +# Add any paths that contain custom themes here, relative to this directory. +#html_theme_path = [] + +# The name for this set of Sphinx documents. If None, it defaults to +# "<project> v<release> documentation". +#html_title = None + +# A shorter title for the navigation bar. Default is the same as html_title. +#html_short_title = None + +# The name of an image file (relative to this directory) to place at the top +# of the sidebar. +#html_logo = None + +# The name of an image file (within the static path) to use as favicon of the +# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 +# pixels large. +#html_favicon = None + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] + +# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, +# using the given strftime format. +#html_last_updated_fmt = '%b %d, %Y' + +# If true, SmartyPants will be used to convert quotes and dashes to +# typographically correct entities. +#html_use_smartypants = True + +# Custom sidebar templates, maps document names to template names. +#html_sidebars = {} + +# Additional templates that should be rendered to pages, maps page names to +# template names. +#html_additional_pages = {} + +# If false, no module index is generated. +#html_domain_indices = True + +# If false, no index is generated. +#html_use_index = True + +# If true, the index is split into individual pages for each letter. +#html_split_index = False + +# If true, links to the reST sources are added to the pages. +#html_show_sourcelink = True + +# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. +#html_show_sphinx = True + +# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. +#html_show_copyright = True + +# If true, an OpenSearch description file will be output, and all pages will +# contain a <link> tag referring to it. The value of this option must be the +# base URL from which the finished HTML is served. +#html_use_opensearch = '' + +# This is the file name suffix for HTML files (e.g. ".xhtml"). +#html_file_suffix = None + +# Output file base name for HTML help builder. +htmlhelp_basename = 'django-storagesdoc' + + +# -- Options for LaTeX output -------------------------------------------------- + +# The paper size ('letter' or 'a4'). +#latex_paper_size = 'letter' + +# The font size ('10pt', '11pt' or '12pt'). +#latex_font_size = '10pt' + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, author, documentclass [howto/manual]). +latex_documents = [ + ('index', 'django-storages.tex', u'django-storages Documentation', + u'David Larlet, et. al.', 'manual'), +] + +# The name of an image file (relative to this directory) to place at the top of +# the title page. +#latex_logo = None + +# For "manual" documents, if this is true, then toplevel headings are parts, +# not chapters. +#latex_use_parts = False + +# If true, show page references after internal links. +#latex_show_pagerefs = False + +# If true, show URL addresses after external links. +#latex_show_urls = False + +# Additional stuff for the LaTeX preamble. +#latex_preamble = '' + +# Documents to append as an appendix to all manuals. +#latex_appendices = [] + +# If false, no module index is generated. +#latex_domain_indices = True + + +# -- Options for manual page output -------------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + ('index', 'django-storages', u'django-storages Documentation', + [u'David Larlet, et. al.'], 1) +] + + +# -- Options for Epub output --------------------------------------------------- + +# Bibliographic Dublin Core info. +epub_title = u'django-storages' +epub_author = u'David Larlet, et. al.' +epub_publisher = u'David Larlet, et. al.' +epub_copyright = u'2011-2013, David Larlet, et. al.' + +# The language of the text. It defaults to the language option +# or en if the language is not set. +#epub_language = '' + +# The scheme of the identifier. Typical schemes are ISBN or URL. +#epub_scheme = '' + +# The unique identifier of the text. This can be a ISBN number +# or the project homepage. +#epub_identifier = '' + +# A unique identification for the text. +#epub_uid = '' + +# HTML files that should be inserted before the pages created by sphinx. +# The format is a list of tuples containing the path and title. +#epub_pre_files = [] + +# HTML files shat should be inserted after the pages created by sphinx. +# The format is a list of tuples containing the path and title. +#epub_post_files = [] + +# A list of files that should not be packed into the epub file. +#epub_exclude_files = [] + +# The depth of the table of contents in toc.ncx. +#epub_tocdepth = 3 + +# Allow duplicate toc entries. +#epub_tocdup = True diff --git a/docs/index.rst b/docs/index.rst new file mode 100644 index 0000000..fa6e013 --- /dev/null +++ b/docs/index.rst @@ -0,0 +1,49 @@ +django-storages +=============== + +django-storages is a collection of custom storage backends for Django. + +.. toctree:: + :maxdepth: 1 + :glob: + + backends/* + +Installation +************ + +Use pip to install from PyPI:: + + pip install django-storages + +Add ``storages`` to your settings.py file:: + + INSTALLED_APPS = ( + ... + 'storages', + ... + ) + +Each storage backend has its own unique settings you will need to add to your settings.py file. Read the documentation for your storage engine(s) of choice to determine what you need to add. + +Contributing +************ + +To contribute to django-storages `create a fork`_ on bitbucket. Clone your fork, make some changes, and submit a pull request. + +.. _create a fork: https://bitbucket.org/david/django-storages/fork + +Issues +****** + +Use the bitbucket `issue tracker`_ for django-storages to submit bugs, issues, and feature requests. + +.. _issue tracker: https://bitbucket.org/david/django-storages/issues + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` + diff --git a/docs/make.bat b/docs/make.bat new file mode 100644 index 0000000..2001686 --- /dev/null +++ b/docs/make.bat @@ -0,0 +1,170 @@ +@ECHO OFF + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set BUILDDIR=_build +set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% . +if NOT "%PAPER%" == "" ( + set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% +) + +if "%1" == "" goto help + +if "%1" == "help" ( + :help + echo.Please use `make ^<target^>` where ^<target^> is one of + echo. html to make standalone HTML files + echo. dirhtml to make HTML files named index.html in directories + echo. singlehtml to make a single large HTML file + echo. pickle to make pickle files + echo. json to make JSON files + echo. htmlhelp to make HTML files and a HTML help project + echo. qthelp to make HTML files and a qthelp project + echo. devhelp to make HTML files and a Devhelp project + echo. epub to make an epub + echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter + echo. text to make text files + echo. man to make manual pages + echo. changes to make an overview over all changed/added/deprecated items + echo. linkcheck to check all external links for integrity + echo. doctest to run all doctests embedded in the documentation if enabled + goto end +) + +if "%1" == "clean" ( + for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i + del /q /s %BUILDDIR%\* + goto end +) + +if "%1" == "html" ( + %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The HTML pages are in %BUILDDIR%/html. + goto end +) + +if "%1" == "dirhtml" ( + %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. + goto end +) + +if "%1" == "singlehtml" ( + %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. + goto end +) + +if "%1" == "pickle" ( + %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle + if errorlevel 1 exit /b 1 + echo. + echo.Build finished; now you can process the pickle files. + goto end +) + +if "%1" == "json" ( + %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json + if errorlevel 1 exit /b 1 + echo. + echo.Build finished; now you can process the JSON files. + goto end +) + +if "%1" == "htmlhelp" ( + %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp + if errorlevel 1 exit /b 1 + echo. + echo.Build finished; now you can run HTML Help Workshop with the ^ +.hhp project file in %BUILDDIR%/htmlhelp. + goto end +) + +if "%1" == "qthelp" ( + %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp + if errorlevel 1 exit /b 1 + echo. + echo.Build finished; now you can run "qcollectiongenerator" with the ^ +.qhcp project file in %BUILDDIR%/qthelp, like this: + echo.^> qcollectiongenerator %BUILDDIR%\qthelp\django-storages.qhcp + echo.To view the help file: + echo.^> assistant -collectionFile %BUILDDIR%\qthelp\django-storages.ghc + goto end +) + +if "%1" == "devhelp" ( + %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. + goto end +) + +if "%1" == "epub" ( + %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The epub file is in %BUILDDIR%/epub. + goto end +) + +if "%1" == "latex" ( + %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex + if errorlevel 1 exit /b 1 + echo. + echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. + goto end +) + +if "%1" == "text" ( + %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The text files are in %BUILDDIR%/text. + goto end +) + +if "%1" == "man" ( + %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man + if errorlevel 1 exit /b 1 + echo. + echo.Build finished. The manual pages are in %BUILDDIR%/man. + goto end +) + +if "%1" == "changes" ( + %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes + if errorlevel 1 exit /b 1 + echo. + echo.The overview file is in %BUILDDIR%/changes. + goto end +) + +if "%1" == "linkcheck" ( + %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck + if errorlevel 1 exit /b 1 + echo. + echo.Link check complete; look for any errors in the above output ^ +or in %BUILDDIR%/linkcheck/output.txt. + goto end +) + +if "%1" == "doctest" ( + %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest + if errorlevel 1 exit /b 1 + echo. + echo.Testing of doctests in the sources finished, look at the ^ +results in %BUILDDIR%/doctest/output.txt. + goto end +) + +:end diff --git a/examples/cloudfiles_project/__init__.py b/examples/cloudfiles_project/__init__.py new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/examples/cloudfiles_project/__init__.py diff --git a/examples/cloudfiles_project/manage.py b/examples/cloudfiles_project/manage.py new file mode 100755 index 0000000..5e78ea9 --- /dev/null +++ b/examples/cloudfiles_project/manage.py @@ -0,0 +1,11 @@ +#!/usr/bin/env python +from django.core.management import execute_manager +try: + import settings # Assumed to be in the same directory. +except ImportError: + import sys + sys.stderr.write("Error: Can't find the file 'settings.py' in the directory containing %r. It appears you've customized things.\nYou'll have to run django-admin.py, passing it your settings module.\n(If the file settings.py does indeed exist, it's causing an ImportError somehow.)\n" % __file__) + sys.exit(1) + +if __name__ == "__main__": + execute_manager(settings) diff --git a/examples/cloudfiles_project/photos/__init__.py b/examples/cloudfiles_project/photos/__init__.py new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/examples/cloudfiles_project/photos/__init__.py diff --git a/examples/cloudfiles_project/photos/admin.py b/examples/cloudfiles_project/photos/admin.py new file mode 100644 index 0000000..74141c5 --- /dev/null +++ b/examples/cloudfiles_project/photos/admin.py @@ -0,0 +1,4 @@ +from django.contrib import admin +from cloudfiles_project.photos.models import Photo + +admin.site.register(Photo)
\ No newline at end of file diff --git a/examples/cloudfiles_project/photos/models.py b/examples/cloudfiles_project/photos/models.py new file mode 100644 index 0000000..0770a3d --- /dev/null +++ b/examples/cloudfiles_project/photos/models.py @@ -0,0 +1,9 @@ +from django.db import models +from backends.mosso import cloudfiles_upload_to + +class Photo(models.Model): + title = models.CharField(max_length=50) + image = models.ImageField(upload_to=cloudfiles_upload_to) + + def __unicode__(self): + return self.title
\ No newline at end of file diff --git a/examples/cloudfiles_project/settings.py b/examples/cloudfiles_project/settings.py new file mode 100644 index 0000000..6c18122 --- /dev/null +++ b/examples/cloudfiles_project/settings.py @@ -0,0 +1,83 @@ +DEBUG = True +TEMPLATE_DEBUG = DEBUG + +ADMINS = ( + # ('Your Name', 'your_email@domain.com'), +) + +MANAGERS = ADMINS + +CLOUDFILES_USERNAME = 'yourusername' +CLOUDFILES_API_KEY = 'yourapikey' +CLOUDFILES_CONTAINER = 'test-container' +CLOUDFILES_TTL = 600 +DEFAULT_FILE_STORAGE = 'backends.mosso.CloudFilesStorage' + +DATABASE_ENGINE = 'sqlite3' # 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'. +DATABASE_NAME = 'local.db' # Or path to database file if using sqlite3. +DATABASE_USER = '' # Not used with sqlite3. +DATABASE_PASSWORD = '' # Not used with sqlite3. +DATABASE_HOST = '' # Set to empty string for localhost. Not used with sqlite3. +DATABASE_PORT = '' # Set to empty string for default. Not used with sqlite3. + +# Local time zone for this installation. Choices can be found here: +# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name +# although not all choices may be available on all operating systems. +# If running in a Windows environment this must be set to the same as your +# system time zone. +TIME_ZONE = 'America/New_York' + +# Language code for this installation. All choices can be found here: +# http://www.i18nguy.com/unicode/language-identifiers.html +LANGUAGE_CODE = 'en-us' + +SITE_ID = 1 + +# If you set this to False, Django will make some optimizations so as not +# to load the internationalization machinery. +USE_I18N = True + +# Absolute path to the directory that holds media. +# Example: "/home/media/media.lawrence.com/" +MEDIA_ROOT = '' + +# URL that handles the media served from MEDIA_ROOT. Make sure to use a +# trailing slash if there is a path component (optional in other cases). +# Examples: "http://media.lawrence.com", "http://example.com/media/" +MEDIA_URL = '/media/' + +# URL prefix for admin media -- CSS, JavaScript and images. Make sure to use a +# trailing slash. +# Examples: "http://foo.com/media/", "/media/". +ADMIN_MEDIA_PREFIX = '/media/admin/' + +# Make this unique, and don't share it with anybody. +SECRET_KEY = '5-7e5&o20#&@&h8t)7%n4a@)y7s3(jnv)qdd_azqzmo826d_u@' + +# List of callables that know how to import templates from various sources. +TEMPLATE_LOADERS = ( + 'django.template.loaders.filesystem.load_template_source', + 'django.template.loaders.app_directories.load_template_source', +# 'django.template.loaders.eggs.load_template_source', +) + +MIDDLEWARE_CLASSES = ( + 'django.middleware.common.CommonMiddleware', + 'django.contrib.sessions.middleware.SessionMiddleware', + 'django.contrib.auth.middleware.AuthenticationMiddleware', +) + +ROOT_URLCONF = 'cloudfiles_project.urls' + +TEMPLATE_DIRS = ( + 'templates', +) + +INSTALLED_APPS = ( + 'django.contrib.auth', + 'django.contrib.contenttypes', + 'django.contrib.sessions', + 'django.contrib.sites', + 'django.contrib.admin', + 'cloudfiles_project.photos', +) diff --git a/examples/cloudfiles_project/templates/base.html b/examples/cloudfiles_project/templates/base.html new file mode 100644 index 0000000..8a8a137 --- /dev/null +++ b/examples/cloudfiles_project/templates/base.html @@ -0,0 +1,14 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" + "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> + +<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> +<head> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> + <title>django-cumulus example project</title> +</head> + +<body> +{% block content %}{% endblock %} + +</body> +</html> diff --git a/examples/cloudfiles_project/templates/photos/photo_list.html b/examples/cloudfiles_project/templates/photos/photo_list.html new file mode 100644 index 0000000..81a49a5 --- /dev/null +++ b/examples/cloudfiles_project/templates/photos/photo_list.html @@ -0,0 +1,14 @@ +{% extends "base.html" %} + +{% block content %} +<h1>Photos in database</h1> +{% if object_list %} +<ul>{% spaceless %} + {% for object in object_list %} + <li>{{ object.image.url }}</li> + {% endfor %} +{% endspaceless %}</ul> +{% else %} +<p>No photos exist.</p> +{% endif %} +{% endblock %} diff --git a/examples/cloudfiles_project/urls.py b/examples/cloudfiles_project/urls.py new file mode 100644 index 0000000..b566697 --- /dev/null +++ b/examples/cloudfiles_project/urls.py @@ -0,0 +1,21 @@ +from django.conf.urls.defaults import * +from django.conf import settings +from photos.models import Photo + +photo_dict = { + 'queryset': Photo.objects.all() +} + +from django.contrib import admin +admin.autodiscover() + +urlpatterns = patterns('', + (r'^admin/doc/', include('django.contrib.admindocs.urls')), + (r'^admin/(.*)', admin.site.root), + (r'^photos/$', 'django.views.generic.list_detail.object_list', photo_dict), +) + +if settings.DEBUG: + urlpatterns += patterns('', + (r'^media/(?P<path>.*)$', 'django.views.static.serve', {'document_root': settings.MEDIA_ROOT}), + )
\ No newline at end of file diff --git a/examples/libcloud_project/manage.py b/examples/libcloud_project/manage.py new file mode 100644 index 0000000..3e4eedc --- /dev/null +++ b/examples/libcloud_project/manage.py @@ -0,0 +1,14 @@ +#!/usr/bin/env python +from django.core.management import execute_manager +import imp +try: + imp.find_module('settings') # Assumed to be in the same directory. +except ImportError: + import sys + sys.stderr.write("Error: Can't find the file 'settings.py' in the directory containing %r. It appears you've customized things.\nYou'll have to run django-admin.py, passing it your settings module.\n" % __file__) + sys.exit(1) + +import settings + +if __name__ == "__main__": + execute_manager(settings) diff --git a/examples/libcloud_project/settings.py b/examples/libcloud_project/settings.py new file mode 100644 index 0000000..3d7bac9 --- /dev/null +++ b/examples/libcloud_project/settings.py @@ -0,0 +1,160 @@ +# Django settings for libcloud_project project. + +DEBUG = True +TEMPLATE_DEBUG = DEBUG + +ADMINS = ( + # ('Your Name', 'your_email@example.com'), +) + +MANAGERS = ADMINS + +DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.sqlite3', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'. + 'NAME': 'libcloud_project.sqllite', # Or path to database file if using sqlite3. + 'USER': '', # Not used with sqlite3. + 'PASSWORD': '', # Not used with sqlite3. + 'HOST': '', # Set to empty string for localhost. Not used with sqlite3. + 'PORT': '', # Set to empty string for default. Not used with sqlite3. + } +} + +# Local time zone for this installation. Choices can be found here: +# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name +# although not all choices may be available on all operating systems. +# On Unix systems, a value of None will cause Django to use the same +# timezone as the operating system. +# If running in a Windows environment this must be set to the same as your +# system time zone. +TIME_ZONE = 'America/Chicago' + +# Language code for this installation. All choices can be found here: +# http://www.i18nguy.com/unicode/language-identifiers.html +LANGUAGE_CODE = 'en-us' + +SITE_ID = 1 + +# If you set this to False, Django will make some optimizations so as not +# to load the internationalization machinery. +USE_I18N = True + +# If you set this to False, Django will not format dates, numbers and +# calendars according to the current locale +USE_L10N = True + +# Absolute filesystem path to the directory that will hold user-uploaded files. +# Example: "/home/media/media.lawrence.com/media/" +MEDIA_ROOT = '' + +# URL that handles the media served from MEDIA_ROOT. Make sure to use a +# trailing slash. +# Examples: "http://media.lawrence.com/media/", "http://example.com/media/" +MEDIA_URL = '' + +# Absolute path to the directory static files should be collected to. +# Don't put anything in this directory yourself; store your static files +# in apps' "static/" subdirectories and in STATICFILES_DIRS. +# Example: "/home/media/media.lawrence.com/static/" +STATIC_ROOT = '' + +# URL prefix for static files. +# Example: "http://media.lawrence.com/static/" +STATIC_URL = '/static/' + +# URL prefix for admin static files -- CSS, JavaScript and images. +# Make sure to use a trailing slash. +# Examples: "http://foo.com/static/admin/", "/static/admin/". +ADMIN_MEDIA_PREFIX = '/static/admin/' + +# Additional locations of static files +STATICFILES_DIRS = ( + # Put strings here, like "/home/html/static" or "C:/www/django/static". + # Always use forward slashes, even on Windows. + # Don't forget to use absolute paths, not relative paths. +) + +# List of finder classes that know how to find static files in +# various locations. +STATICFILES_FINDERS = ( + 'django.contrib.staticfiles.finders.FileSystemFinder', + 'django.contrib.staticfiles.finders.AppDirectoriesFinder', +# 'django.contrib.staticfiles.finders.DefaultStorageFinder', +) + +# Make this unique, and don't share it with anybody. +SECRET_KEY = 'tdzq9m9k-u*k=furpki(@wejb&^2!ea4*z1^t9waj&$)(+$4(h' + +# List of callables that know how to import templates from various sources. +TEMPLATE_LOADERS = ( + 'django.template.loaders.filesystem.Loader', + 'django.template.loaders.app_directories.Loader', +# 'django.template.loaders.eggs.Loader', +) + +MIDDLEWARE_CLASSES = ( + 'django.middleware.common.CommonMiddleware', + 'django.contrib.sessions.middleware.SessionMiddleware', + 'django.middleware.csrf.CsrfViewMiddleware', + 'django.contrib.auth.middleware.AuthenticationMiddleware', + 'django.contrib.messages.middleware.MessageMiddleware', +) + +ROOT_URLCONF = 'libcloud_project.urls' + +TEMPLATE_DIRS = ( + # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates". + # Always use forward slashes, even on Windows. + # Don't forget to use absolute paths, not relative paths. +) + +INSTALLED_APPS = ( + 'django.contrib.auth', + 'django.contrib.contenttypes', + 'django.contrib.sessions', + 'django.contrib.sites', + 'django.contrib.messages', + 'django.contrib.staticfiles', + # Uncomment the next line to enable the admin: + # 'django.contrib.admin', + # Uncomment the next line to enable admin documentation: + # 'django.contrib.admindocs', +) + +# A sample logging configuration. The only tangible logging +# performed by this configuration is to send an email to +# the site admins on every HTTP 500 error. +# See http://docs.djangoproject.com/en/dev/topics/logging for +# more details on how to customize your logging configuration. +LOGGING = { + 'version': 1, + 'disable_existing_loggers': False, + 'handlers': { + 'mail_admins': { + 'level': 'ERROR', + 'class': 'django.utils.log.AdminEmailHandler' + } + }, + 'loggers': { + 'django.request': { + 'handlers': ['mail_admins'], + 'level': 'ERROR', + 'propagate': True, + }, + } +} + +# +# Specific project configuration +# +from libcloud.storage.types import Provider +LIBCLOUD_PROVIDERS = { + 'test_google_storage': { + 'type': Provider.GOOGLE_STORAGE, + 'user': '<google apiv1 user (20 char)>', + 'key': '<google apiv1 key>', + 'bucket': '<bucket name>' + } +} + +DEFAULT_FILE_STORAGE = 'backends.backends.LibCloudStorage'
\ No newline at end of file diff --git a/examples/libcloud_project/test_storage.py b/examples/libcloud_project/test_storage.py new file mode 100644 index 0000000..18d1b72 --- /dev/null +++ b/examples/libcloud_project/test_storage.py @@ -0,0 +1,14 @@ +import sys, os +PROJECT_PATH = os.path.dirname(os.path.abspath(__file__)) +sys.path.append(PROJECT_PATH) + +from django.core.management import setup_environ +import settings +setup_environ(settings) + +from storages.backends.apache_libcloud import LibCloudStorage + +# test_google_storage is a key in settings LIBCLOUD_PROVIDERS dict +store = LibCloudStorage('test_google_storage') +# store is your django storage object that will use google storage +# bucket specified in configuration
\ No newline at end of file diff --git a/examples/libcloud_project/urls.py b/examples/libcloud_project/urls.py new file mode 100644 index 0000000..3eaf5ab --- /dev/null +++ b/examples/libcloud_project/urls.py @@ -0,0 +1,17 @@ +from django.conf.urls.defaults import patterns, include, url + +# Uncomment the next two lines to enable the admin: +# from django.contrib import admin +# admin.autodiscover() + +urlpatterns = patterns('', + # Examples: + # url(r'^$', 'libcloud_project.views.home', name='home'), + # url(r'^libcloud_project/', include('libcloud_project.foo.urls')), + + # Uncomment the admin/doc line below to enable admin documentation: + # url(r'^admin/doc/', include('django.contrib.admindocs.urls')), + + # Uncomment the next line to enable the admin: + # url(r'^admin/', include(admin.site.urls)), +) diff --git a/examples/s3project/__init__.py b/examples/s3project/__init__.py new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/examples/s3project/__init__.py diff --git a/examples/s3project/manage.py b/examples/s3project/manage.py new file mode 100644 index 0000000..77c9127 --- /dev/null +++ b/examples/s3project/manage.py @@ -0,0 +1,15 @@ +# put patched django and S3 in PYTHONPATH +import sys, os +sys.path = [os.path.join(os.getcwd(), '../../')] + sys.path + +from django.core.management import execute_manager + +try: + import settings # Assumed to be in the same directory. +except ImportError: + import sys + sys.stderr.write("Error: Can't find the file 'settings.py' in the directory containing %r. It appears you've customized things.\nYou'll have to run django-admin.py, passing it your settings module.\n(If the file settings.py does indeed exist, it's causing an ImportError somehow.)\n" % __file__) + sys.exit(1) + +if __name__ == "__main__": + execute_manager(settings) diff --git a/examples/s3project/models.py b/examples/s3project/models.py new file mode 100644 index 0000000..0a4815d --- /dev/null +++ b/examples/s3project/models.py @@ -0,0 +1,25 @@ + +import tempfile + +from django.db import models +from django.core.files.base import ContentFile +from django.core.files.storage import default_storage as s3_storage +from django.core.cache import cache + +# Write out a file to be used as default content +s3_storage.save('tests/default.txt', ContentFile('default content')) + +class MyStorage(models.Model): + def custom_upload_to(self, filename): + return 'foo' + + def random_upload_to(self, filename): + # This returns a different result each time, + # to make sure it only gets called once. + import random + return '%s/%s' % (random.randint(100, 999), filename) + + normal = models.FileField(storage=s3_storage, upload_to='tests') + custom = models.FileField(storage=s3_storage, upload_to=custom_upload_to) + random = models.FileField(storage=s3_storage, upload_to=random_upload_to) + default = models.FileField(storage=s3_storage, upload_to='tests', default='tests/default.txt') diff --git a/examples/s3project/settings.py b/examples/s3project/settings.py new file mode 100644 index 0000000..3ac8133 --- /dev/null +++ b/examples/s3project/settings.py @@ -0,0 +1,52 @@ +import os +ROOT_PATH = os.path.dirname(__file__) + +TEMPLATE_DEBUG = DEBUG = True +MANAGERS = ADMINS = () +DATABASE_ENGINE = 'sqlite3' +DATABASE_NAME = os.path.join(ROOT_PATH, 'testdb.sqlite') + +TIME_ZONE = 'America/Chicago' +LANGUAGE_CODE = 'en-us' +SITE_ID = 1 +USE_I18N = True +MEDIA_ROOT = '' +MEDIA_URL = '' +ADMIN_MEDIA_PREFIX = '/media/' +SECRET_KEY = '2+@4vnr#v8e273^+a)g$8%dre^dwcn#d&n#8+l6jk7r#$p&3zk' +TEMPLATE_LOADERS = ( + 'django.template.loaders.filesystem.load_template_source', + 'django.template.loaders.app_directories.load_template_source', +) +MIDDLEWARE_CLASSES = ( + 'django.middleware.common.CommonMiddleware', + 'django.contrib.sessions.middleware.SessionMiddleware', + 'django.contrib.auth.middleware.AuthenticationMiddleware', +) +ROOT_URLCONF = 'urls' +TEMPLATE_DIRS = (os.path.join(ROOT_PATH, 'templates'),) +INSTALLED_APPS = ( + 's3project', # ugly but easier + 'django.contrib.auth', + 'django.contrib.contenttypes', + 'django.contrib.sessions', + 'django.contrib.sites', +) + +DEFAULT_FILE_STORAGE = 'backends.S3Storage.S3Storage' + +from S3 import CallingFormat +AWS_CALLING_FORMAT = CallingFormat.SUBDOMAIN +AWS_HEADERS = { + 'Expires': 'Thu, 15 Apr 2010 20:00:00 GMT', # see http://developer.yahoo.com/performance/rules.html#expires + 'Cache-Control': 'max-age=86400', + } + +# local_settings.py can be used to override environment-specific settings +# like database and email that differ between development and production. +# Add you custom AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and +# AWS_STORAGE_BUCKET_NAME settings +try: + from local_settings import * +except ImportError: + pass diff --git a/examples/s3project/tests.py b/examples/s3project/tests.py new file mode 100644 index 0000000..d849b20 --- /dev/null +++ b/examples/s3project/tests.py @@ -0,0 +1,200 @@ +""" +================= +Django S3 storage +================= + +Usage +===== + +Settings +-------- + +``DEFAULT_FILE_STORAGE`` +~~~~~~~~~~~~~~~~~~~~~~~~ + +This setting store the path to the S3 storage class, the first part correspond +to the filepath and the second the name of the class, if you've got +``example.com`` in your ``PYTHONPATH`` and store your storage file in +``example.com/libs/storages/S3Storage.py``, the resulting setting will be:: + + DEFAULT_FILE_STORAGE = 'libs.storages.S3Storage.S3Storage' + +If you keep the same filename as in repository, it should always end with +``S3Storage.S3Storage``. + +``AWS_ACCESS_KEY_ID`` +~~~~~~~~~~~~~~~~~~~~~ + +Your Amazon Web Services access key, as a string. + +``AWS_SECRET_ACCESS_KEY`` +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Your Amazon Web Services secret access key, as a string. + +``AWS_STORAGE_BUCKET_NAME`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Your Amazon Web Services storage bucket name, as a string. + +``AWS_CALLING_FORMAT`` +~~~~~~~~~~~~~~~~~~~~~~ + +The way you'd like to call the Amazon Web Services API, for instance if you +prefer subdomains:: + + from S3 import CallingFormat + AWS_CALLING_FORMAT = CallingFormat.SUBDOMAIN + +``AWS_HEADERS`` (optionnal) +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you'd like to set headers sent with each file of the storage:: + + # see http://developer.yahoo.com/performance/rules.html#expires + AWS_HEADERS = { + 'Expires': 'Thu, 15 Apr 2010 20:00:00 GMT', + 'Cache-Control': 'max-age=86400', + } + + +Fields +------ + +Once you're done, ``default_storage`` will be the S3 storage:: + + >>> from django.core.files.storage import default_storage + >>> print default_storage.__class__ + <class 'backends.S3Storage.S3Storage'> + +This way, if you define a new ``FileField``, it will use the S3 storage:: + + >>> from django.db import models + >>> class Resume(models.Model): + ... pdf = models.FileField(upload_to='pdfs') + ... photos = models.ImageField(upload_to='photos') + ... + >>> resume = Resume() + >>> print resume.pdf.storage + <backends.S3Storage.S3Storage object at ...> + + +Tests +===== + +Initialization:: + + >>> from django.core.files.storage import default_storage + >>> from django.core.files.base import ContentFile + >>> from django.core.cache import cache + >>> from models import MyStorage + +Storage +------- + +Standard file access options are available, and work as expected:: + + >>> default_storage.exists('storage_test') + False + >>> file = default_storage.open('storage_test', 'w') + >>> file.write('storage contents') + >>> file.close() + + >>> default_storage.exists('storage_test') + True + >>> file = default_storage.open('storage_test', 'r') + >>> file.read() + 'storage contents' + >>> file.close() + + >>> default_storage.delete('storage_test') + >>> default_storage.exists('storage_test') + False + +Model +----- + +An object without a file has limited functionality:: + + >>> obj1 = MyStorage() + >>> obj1.normal + <FieldFile: None> + >>> obj1.normal.size + Traceback (most recent call last): + ... + ValueError: The 'normal' attribute has no file associated with it. + +Saving a file enables full functionality:: + + >>> obj1.normal.save('django_test.txt', ContentFile('content')) + >>> obj1.normal + <FieldFile: tests/django_test.txt> + >>> obj1.normal.size + 7 + >>> obj1.normal.read() + 'content' + +Files can be read in a little at a time, if necessary:: + + >>> obj1.normal.open() + >>> obj1.normal.read(3) + 'con' + >>> obj1.normal.read() + 'tent' + >>> '-'.join(obj1.normal.chunks(chunk_size=2)) + 'co-nt-en-t' + +Save another file with the same name:: + + >>> obj2 = MyStorage() + >>> obj2.normal.save('django_test.txt', ContentFile('more content')) + >>> obj2.normal + <FieldFile: tests/django_test_.txt> + >>> obj2.normal.size + 12 + +Push the objects into the cache to make sure they pickle properly:: + + >>> cache.set('obj1', obj1) + >>> cache.set('obj2', obj2) + >>> cache.get('obj2').normal + <FieldFile: tests/django_test_.txt> + +Deleting an object deletes the file it uses, if there are no other objects +still using that file:: + + >>> obj2.delete() + >>> obj2.normal.save('django_test.txt', ContentFile('more content')) + >>> obj2.normal + <FieldFile: tests/django_test_.txt> + +Default values allow an object to access a single file:: + + >>> obj3 = MyStorage.objects.create() + >>> obj3.default + <FieldFile: tests/default.txt> + >>> obj3.default.read() + 'default content' + +But it shouldn't be deleted, even if there are no more objects using it:: + + >>> obj3.delete() + >>> obj3 = MyStorage() + >>> obj3.default.read() + 'default content' + +Verify the fix for #5655, making sure the directory is only determined once:: + + >>> obj4 = MyStorage() + >>> obj4.random.save('random_file', ContentFile('random content')) + >>> obj4.random + <FieldFile: .../random_file> + +Clean up the temporary files:: + + >>> obj1.normal.delete() + >>> obj2.normal.delete() + >>> obj3.default.delete() + >>> obj4.random.delete() + +"""
\ No newline at end of file diff --git a/setup.py b/setup.py new file mode 100644 index 0000000..26c4e4c --- /dev/null +++ b/setup.py @@ -0,0 +1,25 @@ +from setuptools import setup, find_packages +import storages + +setup( + name = 'django-storages', + version = storages.__version__, + packages = find_packages(), + + author = 'David Larlet', + author_email = 'david@larlet.fr', + license = 'BSD', + description = 'Support for many storages (S3, MogileFS, etc) in Django.', + url='http://code.welldev.org/django-storages/', + classifiers = [ + 'Development Status :: 5 - Production/Stable', + 'Environment :: Web Environment', + 'Intended Audience :: Developers', + 'License :: OSI Approved :: BSD License', + 'Operating System :: OS Independent', + 'Programming Language :: Python', + 'Framework :: Django', + ], + test_suite='tests.main', + zip_safe = False, +) diff --git a/storages/__init__.py b/storages/__init__.py new file mode 100644 index 0000000..f0be139 --- /dev/null +++ b/storages/__init__.py @@ -0,0 +1 @@ +__version__ = '1.1.8' diff --git a/storages/backends/__init__.py b/storages/backends/__init__.py new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/storages/backends/__init__.py diff --git a/storages/backends/apache_libcloud.py b/storages/backends/apache_libcloud.py new file mode 100644 index 0000000..a350d65 --- /dev/null +++ b/storages/backends/apache_libcloud.py @@ -0,0 +1,180 @@ +# Django storage using libcloud providers +# Aymeric Barantal (mric at chamal.fr) 2011 +# +import os + +from django.conf import settings +from django.core.files.storage import Storage +from django.core.files.base import File +from django.core.exceptions import ImproperlyConfigured + +try: + from cStringIO import StringIO +except ImportError: + from StringIO import StringIO + + +try: + from libcloud.storage.providers import get_driver + from libcloud.storage.types import ObjectDoesNotExistError, Provider +except ImportError: + raise ImproperlyConfigured("Could not load libcloud") + + +class LibCloudStorage(Storage): + """Django storage derived class using apache libcloud to operate + on supported providers""" + def __init__(self, provider_name=None, option=None): + if provider_name is None: + provider_name = getattr(settings, 'DEFAULT_LIBCLOUD_PROVIDER', 'default') + + self.provider = settings.LIBCLOUD_PROVIDERS.get(provider_name) + if not self.provider: + raise ImproperlyConfigured( + 'LIBCLOUD_PROVIDERS %s not defined or invalid' % provider_name) + extra_kwargs = {} + if 'region' in self.provider: + extra_kwargs['region'] = self.provider['region'] + try: + provider_type = self.provider['type'] + if isinstance(provider_type, basestring): + module_path, tag = provider_type.rsplit('.', 1) + if module_path != 'libcloud.storage.types.Provider': + raise ValueError("Invalid module path") + provider_type = getattr(Provider, tag) + + Driver = get_driver(provider_type) + self.driver = Driver( + self.provider['user'], + self.provider['key'], + **extra_kwargs + ) + except Exception as e: + raise ImproperlyConfigured( + "Unable to create libcloud driver type %s: %s" % \ + (self.provider.get('type'), e)) + self.bucket = self.provider['bucket'] # Limit to one container + + def _get_bucket(self): + """Helper to get bucket object (libcloud container)""" + return self.driver.get_container(self.bucket) + + def _clean_name(self, name): + """Clean name (windows directories)""" + return os.path.normpath(name).replace('\\', '/') + + def _get_object(self, name): + """Get object by its name. Return None if object not found""" + clean_name = self._clean_name(name) + try: + return self.driver.get_object(self.bucket, clean_name) + except ObjectDoesNotExistError: + return None + + def delete(self, name): + """Delete object on remote""" + obj = self._get_object(name) + if obj: + return self.driver.delete_object(obj) + else: + raise Exception('Object to delete does not exists') + + def exists(self, name): + obj = self._get_object(name) + return True if obj else False + + def listdir(self, path='/'): + """Lists the contents of the specified path, + returning a 2-tuple of lists; the first item being + directories, the second item being files. + """ + container = self._get_bucket() + objects = self.driver.list_container_objects(container) + path = self._clean_name(path) + if not path.endswith('/'): + path = "%s/" % path + files = [] + dirs = [] + # TOFIX: better algorithm to filter correctly + # (and not depend on google-storage empty folder naming) + for o in objects: + if path == '/': + if o.name.count('/') == 0: + files.append(o.name) + elif o.name.count('/') == 1: + dir_name = o.name[:o.name.index('/')] + if not dir_name in dirs: + dirs.append(dir_name) + elif o.name.startswith(path): + if o.name.count('/') <= path.count('/'): + # TOFIX : special case for google storage with empty dir + if o.name.endswith('_$folder$'): + name = o.name[:-9] + name = name[len(path):] + dirs.append(name) + else: + name = o.name[len(path):] + files.append(name) + return (dirs, files) + + def size(self, name): + obj = self._get_object(name) + if obj: + return obj.size + else: + return -1 + + def url(self, name): + obj = self._get_object(name) + return self.driver.get_object_cdn_url(obj) + + def _open(self, name, mode='rb'): + remote_file = LibCloudFile(name, self, mode=mode) + return remote_file + + def _read(self, name, start_range=None, end_range=None): + obj = self._get_object(name) + # TOFIX : we should be able to read chunk by chunk + return next(self.driver.download_object_as_stream(obj, obj.size)) + + def _save(self, name, file): + self.driver.upload_object_via_stream(iter(file), self._get_bucket(), name) + return name + + +class LibCloudFile(File): + """File inherited class for libcloud storage objects read and write""" + def __init__(self, name, storage, mode): + self._name = name + self._storage = storage + self._mode = mode + self._is_dirty = False + self.file = StringIO() + self.start_range = 0 + + @property + def size(self): + if not hasattr(self, '_size'): + self._size = self._storage.size(self._name) + return self._size + + def read(self, num_bytes=None): + if num_bytes is None: + args = [] + self.start_range = 0 + else: + args = [self.start_range, self.start_range + num_bytes - 1] + data = self._storage._read(self._name, *args) + self.file = StringIO(data) + return self.file.getvalue() + + def write(self, content): + if 'w' not in self._mode: + raise AttributeError("File was opened for read-only access.") + self.file = StringIO(content) + self._is_dirty = True + + def close(self): + if self._is_dirty: + self._storage._save(self._name, self.file) + self.file.close() diff --git a/storages/backends/azure_storage.py b/storages/backends/azure_storage.py new file mode 100644 index 0000000..96ef082 --- /dev/null +++ b/storages/backends/azure_storage.py @@ -0,0 +1,65 @@ +import os.path + +from django.core.files.base import ContentFile +from django.core.files.storage import Storage +from django.core.exceptions import ImproperlyConfigured + +try: + import azure + import azure.storage +except ImportError: + raise ImproperlyConfigured( + "Could not load Azure bindings. " + "See https://github.com/WindowsAzure/azure-sdk-for-python") + +from storages.utils import setting + + +def clean_name(name): + return os.path.normpath(name).replace("\\", "/") + + +class AzureStorage(Storage): + account_name = setting("AZURE_ACCOUNT_NAME") + account_key = setting("AZURE_ACCOUNT_KEY") + azure_container = setting("AZURE_CONTAINER") + + def __init__(self, *args, **kwargs): + super(AzureStorage, self).__init__(*args, **kwargs) + self._connection = None + + @property + def connection(self): + if self._connection is None: + self._connection = azure.storage.BlobService( + self.account_name, self.account_key) + return self._connection + + def _open(self, name, mode="rb"): + contents = self.connection.get_blob(self.azure_container, name) + return ContentFile(contents) + + def exists(self, name): + try: + self.connection.get_blob_properties( + self.azure_container, name) + except azure.WindowsAzureMissingResourceError: + return False + else: + return True + + def delete(self, name): + self.connection.delete_blob(self.azure_container, name) + + def size(self, name): + properties = self.connection.get_blob_properties( + self.azure_container, name) + return properties["content-length"] + + def _save(self, name, content): + self.connection.put_blob(self.azure_container, name, + content, "BlockBlob") + return name + + def url(self, name): + return "%s/%s" % (self.azure_bucket, name) diff --git a/storages/backends/couchdb.py b/storages/backends/couchdb.py new file mode 100644 index 0000000..1e116da --- /dev/null +++ b/storages/backends/couchdb.py @@ -0,0 +1,135 @@ +""" +This is a Custom Storage System for Django with CouchDB backend. +Created by Christian Klein. +(c) Copyright 2009 HUDORA GmbH. All Rights Reserved. +""" +import os +from cStringIO import StringIO +from urlparse import urljoin +from urllib import quote_plus + +from django.conf import settings +from django.core.files import File +from django.core.files.storage import Storage +from django.core.exceptions import ImproperlyConfigured + +try: + import couchdb +except ImportError: + raise ImproperlyConfigured("Could not load couchdb dependency.\ + \nSee http://code.google.com/p/couchdb-python/") + +DEFAULT_SERVER= getattr(settings, 'COUCHDB_DEFAULT_SERVER', 'http://couchdb.local:5984') +STORAGE_OPTIONS= getattr(settings, 'COUCHDB_STORAGE_OPTIONS', {}) + + +class CouchDBStorage(Storage): + """ + CouchDBStorage - a Django Storage class for CouchDB. + + The CouchDBStorage can be configured in settings.py, e.g.:: + + COUCHDB_STORAGE_OPTIONS = { + 'server': "http://example.org", + 'database': 'database_name' + } + + Alternatively, the configuration can be passed as a dictionary. + """ + def __init__(self, **kwargs): + kwargs.update(STORAGE_OPTIONS) + self.base_url = kwargs.get('server', DEFAULT_SERVER) + server = couchdb.client.Server(self.base_url) + self.db = server[kwargs.get('database')] + + def _put_file(self, name, content): + self.db[name] = {'size': len(content)} + self.db.put_attachment(self.db[name], content, filename='content') + return name + + def get_document(self, name): + return self.db.get(name) + + def _open(self, name, mode='rb'): + couchdb_file = CouchDBFile(name, self, mode=mode) + return couchdb_file + + def _save(self, name, content): + content.open() + if hasattr(content, 'chunks'): + content_str = ''.join(chunk for chunk in content.chunks()) + else: + content_str = content.read() + name = name.replace('/', '-') + return self._put_file(name, content_str) + + def exists(self, name): + return name in self.db + + def size(self, name): + doc = self.get_document(name) + if doc: + return doc['size'] + return 0 + + def url(self, name): + return urljoin(self.base_url, + os.path.join(quote_plus(self.db.name), + quote_plus(name), + 'content')) + + def delete(self, name): + try: + del self.db[name] + except couchdb.client.ResourceNotFound: + raise IOError("File not found: %s" % name) + + #def listdir(self, name): + # _all_docs? + # pass + + +class CouchDBFile(File): + """ + CouchDBFile - a Django File-like class for CouchDB documents. + """ + + def __init__(self, name, storage, mode): + self._name = name + self._storage = storage + self._mode = mode + self._is_dirty = False + + try: + self._doc = self._storage.get_document(name) + + tmp, ext = os.path.split(name) + if ext: + filename = "content." + ext + else: + filename = "content" + attachment = self._storage.db.get_attachment(self._doc, filename=filename) + self.file = StringIO(attachment) + except couchdb.client.ResourceNotFound: + if 'r' in self._mode: + raise ValueError("The file cannot be reopened.") + else: + self.file = StringIO() + self._is_dirty = True + + @property + def size(self): + return self._doc['size'] + + def write(self, content): + if 'w' not in self._mode: + raise AttributeError("File was opened for read-only access.") + self.file = StringIO(content) + self._is_dirty = True + + def close(self): + if self._is_dirty: + self._storage._put_file(self._name, self.file.getvalue()) + self.file.close() + + diff --git a/storages/backends/database.py b/storages/backends/database.py new file mode 100644 index 0000000..3dd1259 --- /dev/null +++ b/storages/backends/database.py @@ -0,0 +1,132 @@ +# DatabaseStorage for django. +# 2009 (c) GameKeeper Gambling Ltd, Ivanov E. +import StringIO +import urlparse + +from django.conf import settings +from django.core.files import File +from django.core.files.storage import Storage +from django.core.exceptions import ImproperlyConfigured + +try: + import pyodbc +except ImportError: + raise ImproperlyConfigured("Could not load pyodbc dependency.\ + \nSee http://code.google.com/p/pyodbc/") + +REQUIRED_FIELDS = ('db_table', 'fname_column', 'blob_column', 'size_column', 'base_url') + +class DatabaseStorage(Storage): + """ + Class DatabaseStorage provides storing files in the database. + """ + + def __init__(self, option=settings.DB_FILES): + """Constructor. + + Constructs object using dictionary either specified in contucotr or +in settings.DB_FILES. + + @param option dictionary with 'db_table', 'fname_column', +'blob_column', 'size_column', 'base_url' keys. + + option['db_table'] + Table to work with. + option['fname_column'] + Column in the 'db_table' containing filenames (filenames can +contain pathes). Values should be the same as where FileField keeps +filenames. + It is used to map filename to blob_column. In sql it's simply +used in where clause. + option['blob_column'] + Blob column (for example 'image' type), created manually in the +'db_table', used to store image. + option['size_column'] + Column to store file size. Used for optimization of size() +method (another way is to open file and get size) + option['base_url'] + Url prefix used with filenames. Should be mapped to the view, +that returns an image as result. + """ + + if not option or not all([field in option for field in REQUIRED_FIELDS]): + raise ValueError("You didn't specify required options") + + self.db_table = option['db_table'] + self.fname_column = option['fname_column'] + self.blob_column = option['blob_column'] + self.size_column = option['size_column'] + self.base_url = option['base_url'] + + #get database settings + self.DATABASE_ODBC_DRIVER = settings.DATABASE_ODBC_DRIVER + self.DATABASE_NAME = settings.DATABASE_NAME + self.DATABASE_USER = settings.DATABASE_USER + self.DATABASE_PASSWORD = settings.DATABASE_PASSWORD + self.DATABASE_HOST = settings.DATABASE_HOST + + self.connection = pyodbc.connect('DRIVER=%s;SERVER=%s;DATABASE=%s;UID=%s;PWD=%s'%(self.DATABASE_ODBC_DRIVER,self.DATABASE_HOST,self.DATABASE_NAME, + self.DATABASE_USER, self.DATABASE_PASSWORD) ) + self.cursor = self.connection.cursor() + + def _open(self, name, mode='rb'): + """Open a file from database. + + @param name filename or relative path to file based on base_url. path should contain only "/", but not "\". Apache sends pathes with "/". + If there is no such file in the db, returs None + """ + + assert mode == 'rb', "You've tried to open binary file without specifying binary mode! You specified: %s"%mode + + row = self.cursor.execute("SELECT %s from %s where %s = '%s'"%(self.blob_column,self.db_table,self.fname_column,name) ).fetchone() + if row is None: + return None + inMemFile = StringIO.StringIO(row[0]) + inMemFile.name = name + inMemFile.mode = mode + + retFile = File(inMemFile) + return retFile + + def _save(self, name, content): + """Save 'content' as file named 'name'. + + @note '\' in path will be converted to '/'. + """ + + name = name.replace('\\', '/') + binary = pyodbc.Binary(content.read()) + size = len(binary) + + #todo: check result and do something (exception?) if failed. + if self.exists(name): + self.cursor.execute("UPDATE %s SET %s = ?, %s = ? WHERE %s = '%s'"%(self.db_table,self.blob_column,self.size_column,self.fname_column,name), + (binary, size) ) + else: + self.cursor.execute("INSERT INTO %s VALUES(?, ?, ?)"%(self.db_table), (name, binary, size) ) + self.connection.commit() + return name + + def exists(self, name): + row = self.cursor.execute("SELECT %s from %s where %s = '%s'"%(self.fname_column,self.db_table,self.fname_column,name)).fetchone() + return row is not None + + def get_available_name(self, name): + return name + + def delete(self, name): + if self.exists(name): + self.cursor.execute("DELETE FROM %s WHERE %s = '%s'"%(self.db_table,self.fname_column,name)) + self.connection.commit() + + def url(self, name): + if self.base_url is None: + raise ValueError("This file is not accessible via a URL.") + return urlparse.urljoin(self.base_url, name).replace('\\', '/') + + def size(self, name): + row = self.cursor.execute("SELECT %s from %s where %s = '%s'"%(self.size_column,self.db_table,self.fname_column,name)).fetchone() + if row is None: + return 0 + else: + return int(row[0]) diff --git a/storages/backends/ftp.py b/storages/backends/ftp.py new file mode 100644 index 0000000..0012444 --- /dev/null +++ b/storages/backends/ftp.py @@ -0,0 +1,261 @@ +# FTP storage class for Django pluggable storage system. +# Author: Rafal Jonca <jonca.rafal@gmail.com> +# License: MIT +# Comes from http://www.djangosnippets.org/snippets/1269/ +# +# Usage: +# +# Add below to settings.py: +# FTP_STORAGE_LOCATION = '[a]ftp://<user>:<pass>@<host>:<port>/[path]' +# +# In models.py you can write: +# from FTPStorage import FTPStorage +# fs = FTPStorage() +# class FTPTest(models.Model): +# file = models.FileField(upload_to='a/b/c/', storage=fs) + +import os +import ftplib +import urlparse + +try: + from cStringIO import StringIO +except ImportError: + from StringIO import StringIO + +from django.conf import settings +from django.core.files.base import File +from django.core.files.storage import Storage +from django.core.exceptions import ImproperlyConfigured + + +class FTPStorageException(Exception): + pass + + +class FTPStorage(Storage): + """FTP Storage class for Django pluggable storage system.""" + + def __init__(self, location=settings.FTP_STORAGE_LOCATION, + base_url=settings.MEDIA_URL): + self._config = self._decode_location(location) + self._base_url = base_url + self._connection = None + + def _decode_location(self, location): + """Return splitted configuration data from location.""" + splitted_url = urlparse.urlparse(location) + config = {} + + if splitted_url.scheme not in ('ftp', 'aftp'): + raise ImproperlyConfigured( + 'FTPStorage works only with FTP protocol!' + ) + if splitted_url.hostname == '': + raise ImproperlyConfigured('You must at least provide hostname!') + + if splitted_url.scheme == 'aftp': + config['active'] = True + else: + config['active'] = False + config['path'] = splitted_url.path + config['host'] = splitted_url.hostname + config['user'] = splitted_url.username + config['passwd'] = splitted_url.password + config['port'] = int(splitted_url.port) + + return config + + def _start_connection(self): + # Check if connection is still alive and if not, drop it. + if self._connection is not None: + try: + self._connection.pwd() + except ftplib.all_errors: + self._connection = None + + # Real reconnect + if self._connection is None: + ftp = ftplib.FTP() + try: + ftp.connect(self._config['host'], self._config['port']) + ftp.login(self._config['user'], self._config['passwd']) + if self._config['active']: + ftp.set_pasv(False) + if self._config['path'] != '': + ftp.cwd(self._config['path']) + self._connection = ftp + return + except ftplib.all_errors: + raise FTPStorageException( + 'Connection or login error using data %s' + % repr(self._config) + ) + + def disconnect(self): + self._connection.quit() + self._connection = None + + def _mkremdirs(self, path): + pwd = self._connection.pwd() + path_splitted = path.split('/') + for path_part in path_splitted: + try: + self._connection.cwd(path_part) + except: + try: + self._connection.mkd(path_part) + self._connection.cwd(path_part) + except ftplib.all_errors: + raise FTPStorageException( + 'Cannot create directory chain %s' % path + ) + self._connection.cwd(pwd) + return + + def _put_file(self, name, content): + # Connection must be open! + try: + self._mkremdirs(os.path.dirname(name)) + pwd = self._connection.pwd() + self._connection.cwd(os.path.dirname(name)) + self._connection.storbinary('STOR ' + os.path.basename(name), + content.file, + content.DEFAULT_CHUNK_SIZE) + self._connection.cwd(pwd) + except ftplib.all_errors: + raise FTPStorageException('Error writing file %s' % name) + + def _open(self, name, mode='rb'): + remote_file = FTPStorageFile(name, self, mode=mode) + return remote_file + + def _read(self, name): + memory_file = StringIO() + try: + pwd = self._connection.pwd() + self._connection.cwd(os.path.dirname(name)) + self._connection.retrbinary('RETR ' + os.path.basename(name), + memory_file.write) + self._connection.cwd(pwd) + return memory_file + except ftplib.all_errors: + raise FTPStorageException('Error reading file %s' % name) + + def _save(self, name, content): + content.open() + self._start_connection() + self._put_file(name, content) + content.close() + return name + + def _get_dir_details(self, path): + # Connection must be open! + try: + lines = [] + self._connection.retrlines('LIST ' + path, lines.append) + dirs = {} + files = {} + for line in lines: + words = line.split() + if len(words) < 6: + continue + if words[-2] == '->': + continue + if words[0][0] == 'd': + dirs[words[-1]] = 0 + elif words[0][0] == '-': + files[words[-1]] = int(words[-5]) + return dirs, files + except ftplib.all_errors: + raise FTPStorageException('Error getting listing for %s' % path) + + def listdir(self, path): + self._start_connection() + try: + dirs, files = self._get_dir_details(path) + return dirs.keys(), files.keys() + except FTPStorageException: + raise + + def delete(self, name): + if not self.exists(name): + return + self._start_connection() + try: + self._connection.delete(name) + except ftplib.all_errors: + raise FTPStorageException('Error when removing %s' % name) + + def exists(self, name): + self._start_connection() + try: + if os.path.basename(name) in self._connection.nlst( + os.path.dirname(name) + '/' + ): + return True + else: + return False + except ftplib.error_temp: + return False + except ftplib.error_perm: + # error_perm: 550 Can't find file + return False + except ftplib.all_errors: + raise FTPStorageException('Error when testing existence of %s' + % name) + + def size(self, name): + self._start_connection() + try: + dirs, files = self._get_dir_details(os.path.dirname(name)) + if os.path.basename(name) in files: + return files[os.path.basename(name)] + else: + return 0 + except FTPStorageException: + return 0 + + def url(self, name): + if self._base_url is None: + raise ValueError("This file is not accessible via a URL.") + return urlparse.urljoin(self._base_url, name).replace('\\', '/') + + +class FTPStorageFile(File): + def __init__(self, name, storage, mode): + self._name = name + self._storage = storage + self._mode = mode + self._is_dirty = False + self.file = StringIO() + self._is_read = False + + @property + def size(self): + if not hasattr(self, '_size'): + self._size = self._storage.size(self._name) + return self._size + + def read(self, num_bytes=None): + if not self._is_read: + self._storage._start_connection() + self.file = self._storage._read(self._name) + self._storage._end_connection() + self._is_read = True + + return self.file.read(num_bytes) + + def write(self, content): + if 'w' not in self._mode: + raise AttributeError("File was opened for read-only access.") + self.file = StringIO(content) + self._is_dirty = True + self._is_read = True + + def close(self): + if self._is_dirty: + self._storage._start_connection() + self._storage._put_file(self._name, self) + self._storage.disconnect() + self.file.close() diff --git a/storages/backends/gs.py b/storages/backends/gs.py new file mode 100644 index 0000000..d9ef483 --- /dev/null +++ b/storages/backends/gs.py @@ -0,0 +1,99 @@ +try: + from cStringIO import StringIO +except ImportError: + from StringIO import StringIO # noqa + +from django.core.exceptions import ImproperlyConfigured + +from storages.backends.s3boto import S3BotoStorage, S3BotoStorageFile +from storages.utils import setting + +try: + from boto.gs.connection import GSConnection, SubdomainCallingFormat + from boto.exception import GSResponseError + from boto.gs.key import Key as GSKey +except ImportError: + raise ImproperlyConfigured("Could not load Boto's Google Storage bindings.\n" + "See https://github.com/boto/boto") + + +class GSBotoStorageFile(S3BotoStorageFile): + + def write(self, content): + if 'w' not in self._mode: + raise AttributeError("File was not opened in write mode.") + self.file = StringIO(content) + self._is_dirty = True + + def close(self): + if self._is_dirty: + provider = self.key.bucket.connection.provider + upload_headers = {provider.acl_header: self._storage.default_acl} + upload_headers.update(self._storage.headers) + self._storage._save_content(self.key, self.file, upload_headers) + self.key.close() + + +class GSBotoStorage(S3BotoStorage): + connection_class = GSConnection + connection_response_error = GSResponseError + file_class = GSBotoStorageFile + key_class = GSKey + + access_key_names = ['GS_ACCESS_KEY_ID'] + secret_key_names = ['GS_SECRET_ACCESS_KEY'] + + access_key = setting('GS_ACCESS_KEY_ID') + secret_key = setting('GS_SECRET_ACCESS_KEY') + file_overwrite = setting('GS_FILE_OVERWRITE', True) + headers = setting('GS_HEADERS', {}) + bucket_name = setting('GS_BUCKET_NAME', None) + auto_create_bucket = setting('GS_AUTO_CREATE_BUCKET', False) + default_acl = setting('GS_DEFAULT_ACL', 'public-read') + bucket_acl = setting('GS_BUCKET_ACL', default_acl) + querystring_auth = setting('GS_QUERYSTRING_AUTH', True) + querystring_expire = setting('GS_QUERYSTRING_EXPIRE', 3600) + durable_reduced_availability = setting('GS_DURABLE_REDUCED_AVAILABILITY', False) + location = setting('GS_LOCATION', '') + custom_domain = setting('GS_CUSTOM_DOMAIN') + calling_format = setting('GS_CALLING_FORMAT', SubdomainCallingFormat()) + secure_urls = setting('GS_SECURE_URLS', True) + file_name_charset = setting('GS_FILE_NAME_CHARSET', 'utf-8') + is_gzipped = setting('GS_IS_GZIPPED', False) + preload_metadata = setting('GS_PRELOAD_METADATA', False) + gzip_content_types = setting('GS_GZIP_CONTENT_TYPES', ( + 'text/css', + 'application/javascript', + 'application/x-javascript', + )) + url_protocol = setting('GS_URL_PROTOCOL', 'http:') + + def _save_content(self, key, content, headers): + # only pass backwards incompatible arguments if they vary from the default + options = {} + if self.encryption: + options['encrypt_key'] = self.encryption + key.set_contents_from_file(content, headers=headers, + policy=self.default_acl, + rewind=True, **options) + + def _get_or_create_bucket(self, name): + """ + Retrieves a bucket if it exists, otherwise creates it. + """ + if self.durable_reduced_availability: + storage_class = 'DURABLE_REDUCED_AVAILABILITY' + else: + storage_class = 'STANDARD' + try: + return self.connection.get_bucket(name, + validate=self.auto_create_bucket) + except self.connection_response_error: + if self.auto_create_bucket: + bucket = self.connection.create_bucket(name, storage_class=storage_class) + bucket.set_acl(self.bucket_acl) + return bucket + raise ImproperlyConfigured("Bucket %s does not exist. Buckets " + "can be automatically created by " + "setting GS_AUTO_CREATE_BUCKET to " + "``True``." % name) diff --git a/storages/backends/hashpath.py b/storages/backends/hashpath.py new file mode 100644 index 0000000..7603604 --- /dev/null +++ b/storages/backends/hashpath.py @@ -0,0 +1,40 @@ +import os, hashlib, errno + +from django.core.files.storage import FileSystemStorage +from django.utils.encoding import force_unicode + +class HashPathStorage(FileSystemStorage): + """ + Creates a hash from the uploaded file to build the path. + """ + + def save(self, name, content): + # Get the content name if name is not given + if name is None: name = content.name + + # Get the SHA1 hash of the uploaded file + sha1 = hashlib.sha1() + for chunk in content.chunks(): + sha1.update(chunk) + sha1sum = sha1.hexdigest() + + # Build the new path and split it into directory and filename + name = os.path.join(os.path.split(name)[0], sha1sum[:1], sha1sum[1:2], sha1sum) + dir_name, file_name = os.path.split(name) + + # Return the name if the file is already there + if self.exists(name): + return name + + # Try to create the directory relative to location specified in __init__ + try: + os.makedirs(os.path.join(self.location, dir_name)) + except OSError as e: + if e.errno is not errno.EEXIST: + raise e + + # Save the file + name = self._save(name, content) + + # Store filenames with forward slashes, even on Windows + return force_unicode(name.replace('\\', '/')) diff --git a/storages/backends/image.py b/storages/backends/image.py new file mode 100644 index 0000000..ccad24d --- /dev/null +++ b/storages/backends/image.py @@ -0,0 +1,55 @@ + +import os + +from django.core.files.storage import FileSystemStorage +from django.core.exceptions import ImproperlyConfigured + +try: + from PIL import ImageFile as PILImageFile +except ImportError: + raise ImproperlyConfigured("Could not load PIL dependency.\ + \nSee http://www.pythonware.com/products/pil/") + + +class ImageStorage(FileSystemStorage): + """ + A FileSystemStorage which normalizes extensions for images. + + Comes from http://www.djangosnippets.org/snippets/965/ + """ + + def find_extension(self, format): + """Normalizes PIL-returned format into a standard, lowercase extension.""" + format = format.lower() + + if format == 'jpeg': + format = 'jpg' + + return format + + def save(self, name, content): + dirname = os.path.dirname(name) + basename = os.path.basename(name) + + # Use PIL to determine filetype + + p = PILImageFile.Parser() + while 1: + data = content.read(1024) + if not data: + break + p.feed(data) + if p.image: + im = p.image + break + + extension = self.find_extension(im.format) + + # Does the basename already have an extension? If so, replace it. + # bare as in without extension + bare_basename, _ = os.path.splitext(basename) + basename = bare_basename + '.' + extension + + name = os.path.join(dirname, basename) + return super(ImageStorage, self).save(name, content) + diff --git a/storages/backends/mogile.py b/storages/backends/mogile.py new file mode 100644 index 0000000..e609ec7 --- /dev/null +++ b/storages/backends/mogile.py @@ -0,0 +1,118 @@ +from __future__ import print_function + +import urlparse +import mimetypes +from StringIO import StringIO + +from django.conf import settings +from django.core.cache import cache +from django.utils.text import force_unicode +from django.core.files.storage import Storage +from django.http import HttpResponse, HttpResponseNotFound +from django.core.exceptions import ImproperlyConfigured + +try: + import mogilefs +except ImportError: + raise ImproperlyConfigured("Could not load mogilefs dependency.\ + \nSee http://mogilefs.pbworks.com/Client-Libraries") + + +class MogileFSStorage(Storage): + """MogileFS filesystem storage""" + def __init__(self, base_url=settings.MEDIA_URL): + + # the MOGILEFS_MEDIA_URL overrides MEDIA_URL + if hasattr(settings, 'MOGILEFS_MEDIA_URL'): + self.base_url = settings.MOGILEFS_MEDIA_URL + else: + self.base_url = base_url + + for var in ('MOGILEFS_TRACKERS', 'MOGILEFS_DOMAIN',): + if not hasattr(settings, var): + raise ImproperlyConfigured("You must define %s to use the MogileFS backend." % var) + + self.trackers = settings.MOGILEFS_TRACKERS + self.domain = settings.MOGILEFS_DOMAIN + self.client = mogilefs.Client(self.domain, self.trackers) + + def get_mogile_paths(self, filename): + return self.client.get_paths(filename) + + # The following methods define the Backend API + + def filesize(self, filename): + raise NotImplemented + #return os.path.getsize(self._get_absolute_path(filename)) + + def path(self, filename): + paths = self.get_mogile_paths(filename) + if paths: + return self.get_mogile_paths(filename)[0] + else: + return None + + def url(self, filename): + return urlparse.urljoin(self.base_url, filename).replace('\\', '/') + + def open(self, filename, mode='rb'): + raise NotImplemented + #return open(self._get_absolute_path(filename), mode) + + def exists(self, filename): + return filename in self.client + + def save(self, filename, raw_contents): + filename = self.get_available_filename(filename) + + if not hasattr(self, 'mogile_class'): + self.mogile_class = None + + # Write the file to mogile + success = self.client.send_file(filename, StringIO(raw_contents), self.mogile_class) + if success: + print("Wrote file to key %s, %s@%s" % (filename, self.domain, self.trackers[0])) + else: + print("FAILURE writing file %s" % (filename)) + + return force_unicode(filename.replace('\\', '/')) + + def delete(self, filename): + + self.client.delete(filename) + + +def serve_mogilefs_file(request, key=None): + """ + Called when a user requests an image. + Either reproxy the path to perlbal, or serve the image outright + """ + # not the best way to do this, since we create a client each time + mimetype = mimetypes.guess_type(key)[0] or "application/x-octet-stream" + client = mogilefs.Client(settings.MOGILEFS_DOMAIN, settings.MOGILEFS_TRACKERS) + if hasattr(settings, "SERVE_WITH_PERLBAL") and settings.SERVE_WITH_PERLBAL: + # we're reproxying with perlbal + + # check the path cache + + path = cache.get(key) + + if not path: + path = client.get_paths(key) + cache.set(key, path, 60) + + if path: + response = HttpResponse(content_type=mimetype) + response['X-REPROXY-URL'] = path[0] + else: + response = HttpResponseNotFound() + + else: + # we don't have perlbal, let's just serve the image via django + file_data = client[key] + if file_data: + response = HttpResponse(file_data, mimetype=mimetype) + else: + response = HttpResponseNotFound() + + return response diff --git a/storages/backends/mongodb.py b/storages/backends/mongodb.py new file mode 100644 index 0000000..b963729 --- /dev/null +++ b/storages/backends/mongodb.py @@ -0,0 +1,104 @@ +from django.conf import settings +from django.core.exceptions import ImproperlyConfigured +from django.core.files.base import File +from django.core.files.storage import Storage +from django.db import connections +from django.utils.encoding import force_unicode + +try: + from gridfs import GridFS, NoFile +except ImportError: + raise ImproperlyConfigured("Could not load gridfs dependency.\ + \nSee http://www.mongodb.org/display/DOCS/GridFS") + +try: + from pymongo import Connection +except ImportError: + raise ImproperlyConfigured("Could not load pymongo dependency.\ + \nSee http://github.com/mongodb/mongo-python-driver") + +class GridFSStorage(Storage): + @property + def fs(self): + db = settings.GRIDFS_DATABASE + # This should support both the django_mongodb_engine and the GSoC 2010 + # MongoDB backend + from django_mongodb_engine import __version__ + if __version__[0] == 0 and __version__[1] <= 3: + try: + connection = connections[db].db_connection + except: + connection = connections[db].connection + return GridFS(connection) + else: + return GridFS(connections[db].database) + + def _open(self, name, mode='rb'): + return GridFSFile(name, self, mode=mode) + + def _save(self, name, content): + name = force_unicode(name).replace('\\', '/') + content.open() + kwargs = {'filename': name} + if hasattr(content.file, 'content_type'): + kwargs['content_type'] = content.file.content_type + file = self.fs.new_file(**kwargs) + if hasattr(content, 'chunks'): + for chunk in content.chunks(): + file.write(chunk) + else: + file.write(content) + file.close() + content.close() + return name + + def get_valid_name(self, name): + return force_unicode(name).strip().replace('\\', '/') + + def delete(self, name): + f = self._open(name, 'r') + return self.fs.delete(f.file._id) + + def exists(self, name): + try: + self.fs.get_last_version(name) + return True + except NoFile: + return False + + def listdir(self, path): + return ((), self.fs.list()) + + def size(self, name): + try: + return self.fs.get_last_version(name).length + except NoFile: + raise ValueError('File with name "%s" does not exist' % name) + + def url(self, name): + raise NotImplementedError() + +class GridFSFile(File): + def __init__(self, name, storage, mode): + self.name = name + self._storage = storage + self._mode = mode + + try: + self.file = storage.fs.get_last_version(name) + except NoFile: + raise ValueError("The file doesn't exist.") + + @property + def size(self): + return self.file.length + + def read(self, num_bytes=None): + return self.file.read(num_bytes) + + def write(self, content): + raise NotImplementedError() + + def close(self): + self.file.close() + diff --git a/storages/backends/mosso.py b/storages/backends/mosso.py new file mode 100644 index 0000000..5a4812d --- /dev/null +++ b/storages/backends/mosso.py @@ -0,0 +1,347 @@ +""" +Custom storage for django with Mosso Cloud Files backend. +Created by Rich Leland <rich@richleland.com>. +""" +import os +import warnings +warnings.simplefilter('always', PendingDeprecationWarning) +warnings.warn("The mosso module will be deprecated in version 1.2 of " + "django-storages. The CloudFiles code has been moved into" + "django-cumulus at http://github.com/richleland/django-cumulus.", + PendingDeprecationWarning) + +from django.conf import settings +from django.core.exceptions import ImproperlyConfigured +from django.core.files import File +from django.core.files.storage import Storage +from django.utils.text import get_valid_filename + +try: + from cStringIO import StringIO +except: + from StringIO import StringIO + +try: + import cloudfiles + from cloudfiles.errors import NoSuchObject +except ImportError: + raise ImproperlyConfigured("Could not load cloudfiles dependency. See " + "http://www.mosso.com/cloudfiles.jsp.") + +# TODO: implement TTL into cloudfiles methods +TTL = getattr(settings, 'CLOUDFILES_TTL', 600) +CONNECTION_KWARGS = getattr(settings, 'CLOUDFILES_CONNECTION_KWARGS', {}) +SSL = getattr(settings, 'CLOUDFILES_SSL', False) + + +def cloudfiles_upload_to(self, filename): + """ + Simple, custom upload_to because Cloud Files doesn't support + nested containers (directories). + + Actually found this out from @minter: + @richleland The Cloud Files APIs do support pseudo-subdirectories, by + creating zero-byte files with type application/directory. + + May implement in a future version. + """ + return get_valid_filename(filename) + + +class CloudFilesStorage(Storage): + """ + Custom storage for Mosso Cloud Files. + """ + default_quick_listdir = True + + def __init__(self, + username=settings.CLOUDFILES_USERNAME, + api_key=settings.CLOUDFILES_API_KEY, + container=settings.CLOUDFILES_CONTAINER, + connection_kwargs=CONNECTION_KWARGS): + """ + Initialize the settings for the connection and container. + """ + self.username = username + self.api_key = api_key + self.container_name = container + self.connection_kwargs = connection_kwargs + + def __getstate__(self): + """ + Return a picklable representation of the storage. + """ + return dict(username=self.username, + api_key=self.api_key, + container_name=self.container_name, + connection_kwargs=self.connection_kwargs) + + def _get_connection(self): + if not hasattr(self, '_connection'): + self._connection = cloudfiles.get_connection(self.username, + self.api_key, **self.connection_kwargs) + return self._connection + + def _set_connection(self, value): + self._connection = value + + connection = property(_get_connection, _set_connection) + + def _get_container(self): + if not hasattr(self, '_container'): + self.container = self.connection.get_container( + self.container_name) + return self._container + + def _set_container(self, container): + """ + Set the container, making it publicly available (on Limelight CDN) if + it is not already. + """ + if not container.is_public(): + container.make_public() + if hasattr(self, '_container_public_uri'): + delattr(self, '_container_public_uri') + self._container = container + + container = property(_get_container, _set_container) + + def _get_container_url(self): + if not hasattr(self, '_container_public_uri'): + if SSL: + self._container_public_uri = self.container.public_ssl_uri() + else: + self._container_public_uri = self.container.public_uri() + return self._container_public_uri + + container_url = property(_get_container_url) + + def _get_cloud_obj(self, name): + """ + Helper function to get retrieve the requested Cloud Files Object. + """ + return self.container.get_object(name) + + def _open(self, name, mode='rb'): + """ + Return the CloudFilesStorageFile. + """ + return CloudFilesStorageFile(storage=self, name=name) + + def _save(self, name, content): + """ + Use the Cloud Files service to write ``content`` to a remote file + (called ``name``). + """ + (path, last) = os.path.split(name) + if path: + try: + self.container.get_object(path) + except NoSuchObject: + self._save(path, CloudStorageDirectory(path)) + + cloud_obj = self.container.create_object(name) + cloud_obj.size = content.size + + content.open() + # If the content type is available, pass it in directly rather than + # getting the cloud object to try to guess. + if hasattr(content.file, 'content_type'): + cloud_obj.content_type = content.file.content_type + cloud_obj.send(content) + content.close() + return name + + def delete(self, name): + """ + Deletes the specified file from the storage system. + """ + # If the file exists, delete it. + if self.exists(name): + self.container.delete_object(name) + + def exists(self, name): + """ + Returns True if a file referenced by the given name already exists in + the storage system, or False if the name is available for a new file. + """ + try: + self._get_cloud_obj(name) + return True + except NoSuchObject: + return False + + def listdir(self, path): + """ + Lists the contents of the specified path, returning a 2-tuple; the + first being an empty list of directories (not available for quick- + listing), the second being a list of filenames. + + If the list of directories is required, use the full_listdir method. + """ + files = [] + if path and not path.endswith('/'): + path = '%s/' % path + path_len = len(path) + for name in self.container.list_objects(path=path): + files.append(name[path_len:]) + return ([], files) + + def full_listdir(self, path): + """ + Lists the contents of the specified path, returning a 2-tuple of lists; + the first item being directories, the second item being files. + + On large containers, this may be a slow operation for root containers + because every single object must be returned (cloudfiles does not + provide an explicit way of listing directories). + """ + dirs = set() + files = [] + if path and not path.endswith('/'): + path = '%s/' % path + path_len = len(path) + for name in self.container.list_objects(prefix=path): + name = name[path_len:] + slash = name[1:-1].find('/') + 1 + if slash: + dirs.add(name[:slash]) + elif name: + files.append(name) + dirs = list(dirs) + dirs.sort() + return (dirs, files) + + def size(self, name): + """ + Returns the total size, in bytes, of the file specified by name. + """ + return self._get_cloud_obj(name).size + + def url(self, name): + """ + Returns an absolute URL where the file's contents can be accessed + directly by a web browser. + """ + return '%s/%s' % (self.container_url, name) + + +class CloudStorageDirectory(File): + """ + A File-like object that creates a directory at cloudfiles + """ + + def __init__(self, name): + super(CloudStorageDirectory, self).__init__(StringIO(), name=name) + self.file.content_type = 'application/directory' + self.size = 0 + + def __str__(self): + return 'directory' + + def __nonzero__(self): + return True + + def open(self, mode=None): + self.seek(0) + + def close(self): + pass + + +class CloudFilesStorageFile(File): + closed = False + + def __init__(self, storage, name, *args, **kwargs): + self._storage = storage + super(CloudFilesStorageFile, self).__init__(file=None, name=name, + *args, **kwargs) + self._pos = 0 + + + def _get_size(self): + if not hasattr(self, '_size'): + self._size = self._storage.size(self.name) + return self._size + + def _set_size(self, size): + self._size = size + + size = property(_get_size, _set_size) + + def _get_file(self): + if not hasattr(self, '_file'): + self._file = self._storage._get_cloud_obj(self.name) + return self._file + + def _set_file(self, value): + if value is None: + if hasattr(self, '_file'): + del self._file + else: + self._file = value + + file = property(_get_file, _set_file) + + def read(self, num_bytes=None): + if self._pos == self._get_size(): + return None + if self._pos + num_bytes > self._get_size(): + num_bytes = self._get_size() - self._pos + data = self.file.read(size=num_bytes or -1, offset=self._pos) + self._pos += len(data) + return data + + def open(self, *args, **kwargs): + """ + Open the cloud file object. + """ + self.file + self._pos = 0 + + def close(self, *args, **kwargs): + self._pos = 0 + + @property + def closed(self): + return not hasattr(self, '_file') + + def seek(self, pos): + self._pos = pos + + +class ThreadSafeCloudFilesStorage(CloudFilesStorage): + """ + Extends CloudFilesStorage to make it thread safer. + + As long as you don't pass container or cloud objects + between threads, you'll be thread safe. + + Uses one cloudfiles connection per thread. + """ + + def __init__(self, *args, **kwargs): + super(ThreadSafeCloudFilesStorage, self).__init__(*args, **kwargs) + + import threading + self.local_cache = threading.local() + + def _get_connection(self): + if not hasattr(self.local_cache, 'connection'): + connection = cloudfiles.get_connection(self.username, + self.api_key, **self.connection_kwargs) + self.local_cache.connection = connection + + return self.local_cache.connection + + connection = property(_get_connection, CloudFilesStorage._set_connection) + + def _get_container(self): + if not hasattr(self.local_cache, 'container'): + container = self.connection.get_container(self.container_name) + self.local_cache.container = container + + return self.local_cache.container + + container = property(_get_container, CloudFilesStorage._set_container) + diff --git a/storages/backends/overwrite.py b/storages/backends/overwrite.py new file mode 100644 index 0000000..fba464a --- /dev/null +++ b/storages/backends/overwrite.py @@ -0,0 +1,19 @@ +from django.core.files.storage import FileSystemStorage + + +class OverwriteStorage(FileSystemStorage): + """ + Comes from http://www.djangosnippets.org/snippets/976/ + (even if it already exists in S3Storage for ages) + + See also Django #4339, which might add this functionality to core. + """ + + def get_available_name(self, name): + """ + Returns a filename that's free on the target storage system, and + available for new content to be written to. + """ + if self.exists(name): + self.delete(name) + return name diff --git a/storages/backends/s3.py b/storages/backends/s3.py new file mode 100644 index 0000000..a700596 --- /dev/null +++ b/storages/backends/s3.py @@ -0,0 +1,288 @@ +import os +import mimetypes +import warnings + +try: + from cStringIO import StringIO +except ImportError: + from StringIO import StringIO + +from django.conf import settings +from django.core.files.base import File +from django.core.files.storage import Storage +from django.core.exceptions import ImproperlyConfigured + +try: + from S3 import AWSAuthConnection, QueryStringAuthGenerator, CallingFormat +except ImportError: + raise ImproperlyConfigured("Could not load amazon's S3 bindings.\nSee " + "http://developer.amazonwebservices.com/connect/entry.jspa?externalID=134") + +ACCESS_KEY_NAME = getattr(settings, 'AWS_S3_ACCESS_KEY_ID', getattr(settings, 'AWS_ACCESS_KEY_ID', None)) +SECRET_KEY_NAME = getattr(settings, 'AWS_S3_SECRET_ACCESS_KEY', getattr(settings, 'AWS_SECRET_ACCESS_KEY', None)) +HEADERS = getattr(settings, 'AWS_HEADERS', {}) +DEFAULT_ACL = getattr(settings, 'AWS_DEFAULT_ACL', 'public-read') +QUERYSTRING_ACTIVE = getattr(settings, 'AWS_QUERYSTRING_ACTIVE', False) +QUERYSTRING_EXPIRE = getattr(settings, 'AWS_QUERYSTRING_EXPIRE', 60) +SECURE_URLS = getattr(settings, 'AWS_S3_SECURE_URLS', False) +BUCKET_PREFIX = getattr(settings, 'AWS_BUCKET_PREFIX', '') +CALLING_FORMAT = getattr(settings, 'AWS_CALLING_FORMAT', CallingFormat.PATH) +PRELOAD_METADATA = getattr(settings, 'AWS_PRELOAD_METADATA', False) + +IS_GZIPPED = getattr(settings, 'AWS_IS_GZIPPED', False) +GZIP_CONTENT_TYPES = getattr(settings, 'GZIP_CONTENT_TYPES', ( + 'text/css', + 'application/javascript', + 'application/x-javascript' +)) + +if IS_GZIPPED: + from gzip import GzipFile + +class S3Storage(Storage): + """Amazon Simple Storage Service""" + + def __init__(self, bucket=settings.AWS_STORAGE_BUCKET_NAME, + access_key=None, secret_key=None, acl=DEFAULT_ACL, + calling_format=CALLING_FORMAT, encrypt=False, + gzip=IS_GZIPPED, gzip_content_types=GZIP_CONTENT_TYPES, + preload_metadata=PRELOAD_METADATA): + warnings.warn( + "The s3 backend is deprecated and will be removed in version 1.2. " + "Use the s3boto backend instead.", + PendingDeprecationWarning + ) + self.bucket = bucket + self.acl = acl + self.encrypt = encrypt + self.gzip = gzip + self.gzip_content_types = gzip_content_types + self.preload_metadata = preload_metadata + + if encrypt: + try: + import ezPyCrypto + except ImportError: + raise ImproperlyConfigured("Could not load ezPyCrypto.\nSee " + "http://www.freenet.org.nz/ezPyCrypto/ to install it.") + self.crypto_key = ezPyCrypto.key + + if not access_key and not secret_key: + access_key, secret_key = self._get_access_keys() + + self.connection = AWSAuthConnection(access_key, secret_key, + calling_format=calling_format) + self.generator = QueryStringAuthGenerator(access_key, secret_key, + calling_format=calling_format, + is_secure=SECURE_URLS) + self.generator.set_expires_in(QUERYSTRING_EXPIRE) + + self.headers = HEADERS + self._entries = {} + + def _get_access_keys(self): + access_key = ACCESS_KEY_NAME + secret_key = SECRET_KEY_NAME + if (access_key or secret_key) and (not access_key or not secret_key): + access_key = os.environ.get(ACCESS_KEY_NAME) + secret_key = os.environ.get(SECRET_KEY_NAME) + + if access_key and secret_key: + # Both were provided, so use them + return access_key, secret_key + + return None, None + + @property + def entries(self): + if self.preload_metadata and not self._entries: + self._entries = dict((entry.key, entry) + for entry in self.connection.list_bucket(self.bucket).entries) + return self._entries + + def _get_connection(self): + return AWSAuthConnection(*self._get_access_keys()) + + def _clean_name(self, name): + # Useful for windows' paths + return os.path.join(BUCKET_PREFIX, os.path.normpath(name).replace('\\', '/')) + + def _compress_string(self, s): + """Gzip a given string.""" + zbuf = StringIO() + zfile = GzipFile(mode='wb', compresslevel=6, fileobj=zbuf) + zfile.write(s) + zfile.close() + return zbuf.getvalue() + + def _put_file(self, name, content): + if self.encrypt: + + # Create a key object + key = self.crypto_key() + + # Read in a public key + fd = open(settings.CRYPTO_KEYS_PUBLIC, "rb") + public_key = fd.read() + fd.close() + + # import this public key + key.importKey(public_key) + + # Now encrypt some text against this public key + content = key.encString(content) + + content_type = mimetypes.guess_type(name)[0] or "application/x-octet-stream" + + if self.gzip and content_type in self.gzip_content_types: + content = self._compress_string(content) + self.headers.update({'Content-Encoding': 'gzip'}) + + self.headers.update({ + 'x-amz-acl': self.acl, + 'Content-Type': content_type, + 'Content-Length' : str(len(content)), + }) + response = self.connection.put(self.bucket, name, content, self.headers) + if response.http_response.status not in (200, 206): + raise IOError("S3StorageError: %s" % response.message) + + def _open(self, name, mode='rb'): + name = self._clean_name(name) + remote_file = S3StorageFile(name, self, mode=mode) + return remote_file + + def _read(self, name, start_range=None, end_range=None): + name = self._clean_name(name) + if start_range is None: + headers = {} + else: + headers = {'Range': 'bytes=%s-%s' % (start_range, end_range)} + response = self.connection.get(self.bucket, name, headers) + if response.http_response.status not in (200, 206): + raise IOError("S3StorageError: %s" % response.message) + headers = response.http_response.msg + + if self.encrypt: + # Read in a private key + fd = open(settings.CRYPTO_KEYS_PRIVATE, "rb") + private_key = fd.read() + fd.close() + + # Create a key object, and auto-import private key + key = self.crypto_key(private_key) + + # Decrypt this file + response.object.data = key.decString(response.object.data) + + return response.object.data, headers.get('etag', None), headers.get('content-range', None) + + def _save(self, name, content): + name = self._clean_name(name) + content.open() + if hasattr(content, 'chunks'): + content_str = ''.join(chunk for chunk in content.chunks()) + else: + content_str = content.read() + self._put_file(name, content_str) + return name + + def delete(self, name): + name = self._clean_name(name) + response = self.connection.delete(self.bucket, name) + if response.http_response.status != 204: + raise IOError("S3StorageError: %s" % response.message) + + def exists(self, name): + name = self._clean_name(name) + if self.entries: + return name in self.entries + response = self.connection._make_request('HEAD', self.bucket, name) + return response.status == 200 + + def size(self, name): + name = self._clean_name(name) + if self.entries: + entry = self.entries.get(name) + if entry: + return entry.size + return 0 + response = self.connection._make_request('HEAD', self.bucket, name) + content_length = response.getheader('Content-Length') + return content_length and int(content_length) or 0 + + def url(self, name): + name = self._clean_name(name) + if QUERYSTRING_ACTIVE: + return self.generator.generate_url('GET', self.bucket, name) + else: + return self.generator.make_bare_url(self.bucket, name) + + def modified_time(self, name): + try: + from dateutil import parser, tz + except ImportError: + raise NotImplementedError() + name = self._clean_name(name) + if self.entries: + last_modified = self.entries.get(name).last_modified + else: + response = self.connection._make_request('HEAD', self.bucket, name) + last_modified = response.getheader('Last-Modified') + # convert to string to date + last_modified_date = parser.parse(last_modified) + # if the date has no timzone, assume UTC + if last_modified_date.tzinfo == None: + last_modified_date = last_modified_date.replace(tzinfo=tz.tzutc()) + # convert date to local time w/o timezone + return last_modified_date.astimezone(tz.tzlocal()).replace(tzinfo=None) + + ## UNCOMMENT BELOW IF NECESSARY + #def get_available_name(self, name): + # """ Overwrite existing file with the same name. """ + # name = self._clean_name(name) + # return name + + +class PreloadingS3Storage(S3Storage): + pass + +class S3StorageFile(File): + def __init__(self, name, storage, mode): + self._name = name + self._storage = storage + self._mode = mode + self._is_dirty = False + self.file = StringIO() + self.start_range = 0 + + @property + def size(self): + if not hasattr(self, '_size'): + self._size = self._storage.size(self._name) + return self._size + + def read(self, num_bytes=None): + if num_bytes is None: + args = [] + self.start_range = 0 + else: + args = [self.start_range, self.start_range+num_bytes-1] + data, etags, content_range = self._storage._read(self._name, *args) + if content_range is not None: + current_range, size = content_range.split(' ', 1)[1].split('/', 1) + start_range, end_range = current_range.split('-', 1) + self._size, self.start_range = int(size), int(end_range)+1 + self.file = StringIO(data) + return self.file.getvalue() + + def write(self, content): + if 'w' not in self._mode: + raise AttributeError("File was opened for read-only access.") + self.file = StringIO(content) + self._is_dirty = True + + def close(self): + if self._is_dirty: + self._storage._put_file(self._name, self.file.getvalue()) + self.file.close() diff --git a/storages/backends/s3boto.py b/storages/backends/s3boto.py new file mode 100644 index 0000000..354cd0c --- /dev/null +++ b/storages/backends/s3boto.py @@ -0,0 +1,501 @@ +import os +import posixpath +import mimetypes +from gzip import GzipFile +import datetime +from tempfile import SpooledTemporaryFile + +try: + from cStringIO import StringIO +except ImportError: + from StringIO import StringIO # noqa + +from django.core.files.base import File +from django.core.files.storage import Storage +from django.core.exceptions import ImproperlyConfigured, SuspiciousOperation +from django.utils.encoding import force_unicode, smart_str, filepath_to_uri + +try: + from boto import __version__ as boto_version + from boto.s3.connection import S3Connection, SubdomainCallingFormat + from boto.exception import S3ResponseError + from boto.s3.key import Key as S3Key + from boto.utils import parse_ts +except ImportError: + raise ImproperlyConfigured("Could not load Boto's S3 bindings.\n" + "See https://github.com/boto/boto") + +from storages.utils import setting + +boto_version_info = tuple([int(i) for i in boto_version.split('-')[0].split('.')]) + +if boto_version_info[:2] < (2, 4): + raise ImproperlyConfigured("The installed Boto library must be 2.4 or " + "higher.\nSee https://github.com/boto/boto") + + +def parse_ts_extended(ts): + RFC1123 = '%a, %d %b %Y %H:%M:%S %Z' + rv = None + try: + rv = parse_ts(ts) + except ValueError: + rv = datetime.datetime.strptime(ts, RFC1123) + return rv + + +def safe_join(base, *paths): + """ + A version of django.utils._os.safe_join for S3 paths. + + Joins one or more path components to the base path component + intelligently. Returns a normalized version of the final path. + + The final path must be located inside of the base path component + (otherwise a ValueError is raised). + + Paths outside the base path indicate a possible security + sensitive operation. + """ + from urlparse import urljoin + base_path = force_unicode(base) + base_path = base_path.rstrip('/') + paths = [force_unicode(p) for p in paths] + + final_path = base_path + for path in paths: + final_path = urljoin(final_path.rstrip('/') + "/", path) + + # Ensure final_path starts with base_path and that the next character after + # the final path is '/' (or nothing, in which case final_path must be + # equal to base_path). + base_path_len = len(base_path) + if (not final_path.startswith(base_path) or + final_path[base_path_len:base_path_len + 1] not in ('', '/')): + raise ValueError('the joined path is located outside of the base path' + ' component') + + return final_path.lstrip('/') + + +class S3BotoStorageFile(File): + """ + The default file object used by the S3BotoStorage backend. + + This file implements file streaming using boto's multipart + uploading functionality. The file can be opened in read or + write mode. + + This class extends Django's File class. However, the contained + data is only the data contained in the current buffer. So you + should not access the contained file object directly. You should + access the data via this class. + + Warning: This file *must* be closed using the close() method in + order to properly write the file to S3. Be sure to close the file + in your application. + """ + # TODO: Read/Write (rw) mode may be a bit undefined at the moment. Needs testing. + # TODO: When Django drops support for Python 2.5, rewrite to use the + # BufferedIO streams in the Python 2.6 io module. + buffer_size = setting('AWS_S3_FILE_BUFFER_SIZE', 5242880) + + def __init__(self, name, mode, storage, buffer_size=None): + self._storage = storage + self.name = name[len(self._storage.location):].lstrip('/') + self._mode = mode + self.key = storage.bucket.get_key(self._storage._encode_name(name)) + if not self.key and 'w' in mode: + self.key = storage.bucket.new_key(storage._encode_name(name)) + self._is_dirty = False + self._file = None + self._multipart = None + # 5 MB is the minimum part size (if there is more than one part). + # Amazon allows up to 10,000 parts. The default supports uploads + # up to roughly 50 GB. Increase the part size to accommodate + # for files larger than this. + if buffer_size is not None: + self.buffer_size = buffer_size + self._write_counter = 0 + + @property + def size(self): + return self.key.size + + def _get_file(self): + if self._file is None: + self._file = SpooledTemporaryFile( + max_size=self._storage.max_memory_size, + suffix=".S3BotoStorageFile", + dir=setting("FILE_UPLOAD_TEMP_DIR", None) + ) + if 'r' in self._mode: + self._is_dirty = False + self.key.get_contents_to_file(self._file) + self._file.seek(0) + if self._storage.gzip and self.key.content_encoding == 'gzip': + self._file = GzipFile(mode=self._mode, fileobj=self._file) + return self._file + + def _set_file(self, value): + self._file = value + + file = property(_get_file, _set_file) + + def read(self, *args, **kwargs): + if 'r' not in self._mode: + raise AttributeError("File was not opened in read mode.") + return super(S3BotoStorageFile, self).read(*args, **kwargs) + + def write(self, *args, **kwargs): + if 'w' not in self._mode: + raise AttributeError("File was not opened in write mode.") + self._is_dirty = True + if self._multipart is None: + provider = self.key.bucket.connection.provider + upload_headers = { + provider.acl_header: self._storage.default_acl + } + upload_headers.update({'Content-Type': mimetypes.guess_type(self.key.name)[0] or self._storage.key_class.DefaultContentType}) + upload_headers.update(self._storage.headers) + self._multipart = self._storage.bucket.initiate_multipart_upload( + self.key.name, + headers=upload_headers, + reduced_redundancy=self._storage.reduced_redundancy + ) + if self.buffer_size <= self._buffer_file_size: + self._flush_write_buffer() + return super(S3BotoStorageFile, self).write(*args, **kwargs) + + @property + def _buffer_file_size(self): + pos = self.file.tell() + self.file.seek(0, os.SEEK_END) + length = self.file.tell() + self.file.seek(pos) + return length + + def _flush_write_buffer(self): + """ + Flushes the write buffer. + """ + if self._buffer_file_size: + self._write_counter += 1 + self.file.seek(0) + headers = self._storage.headers.copy() + self._multipart.upload_part_from_file( + self.file, self._write_counter, headers=headers) + self.file.close() + self._file = None + + def close(self): + if self._is_dirty: + self._flush_write_buffer() + self._multipart.complete_upload() + else: + if not self._multipart is None: + self._multipart.cancel_upload() + self.key.close() + + +class S3BotoStorage(Storage): + """ + Amazon Simple Storage Service using Boto + + This storage backend supports opening files in read or write + mode and supports streaming(buffering) data in chunks to S3 + when writing. + """ + connection_class = S3Connection + connection_response_error = S3ResponseError + file_class = S3BotoStorageFile + key_class = S3Key + + # used for looking up the access and secret key from env vars + access_key_names = ['AWS_S3_ACCESS_KEY_ID', 'AWS_ACCESS_KEY_ID'] + secret_key_names = ['AWS_S3_SECRET_ACCESS_KEY', 'AWS_SECRET_ACCESS_KEY'] + + access_key = setting('AWS_S3_ACCESS_KEY_ID', setting('AWS_ACCESS_KEY_ID')) + secret_key = setting('AWS_S3_SECRET_ACCESS_KEY', setting('AWS_SECRET_ACCESS_KEY')) + file_overwrite = setting('AWS_S3_FILE_OVERWRITE', True) + headers = setting('AWS_HEADERS', {}) + bucket_name = setting('AWS_STORAGE_BUCKET_NAME') + auto_create_bucket = setting('AWS_AUTO_CREATE_BUCKET', False) + default_acl = setting('AWS_DEFAULT_ACL', 'public-read') + bucket_acl = setting('AWS_BUCKET_ACL', default_acl) + querystring_auth = setting('AWS_QUERYSTRING_AUTH', True) + querystring_expire = setting('AWS_QUERYSTRING_EXPIRE', 3600) + reduced_redundancy = setting('AWS_REDUCED_REDUNDANCY', False) + location = setting('AWS_LOCATION', '') + encryption = setting('AWS_S3_ENCRYPTION', False) + custom_domain = setting('AWS_S3_CUSTOM_DOMAIN') + calling_format = setting('AWS_S3_CALLING_FORMAT', SubdomainCallingFormat()) + secure_urls = setting('AWS_S3_SECURE_URLS', True) + file_name_charset = setting('AWS_S3_FILE_NAME_CHARSET', 'utf-8') + gzip = setting('AWS_IS_GZIPPED', False) + preload_metadata = setting('AWS_PRELOAD_METADATA', False) + gzip_content_types = setting('GZIP_CONTENT_TYPES', ( + 'text/css', + 'application/javascript', + 'application/x-javascript', + )) + url_protocol = setting('AWS_S3_URL_PROTOCOL', 'http:') + host = setting('AWS_S3_HOST', S3Connection.DefaultHost) + use_ssl = setting('AWS_S3_USE_SSL', True) + port = setting('AWS_S3_PORT', None) + + # The max amount of memory a returned file can take up before being + # rolled over into a temporary file on disk. Default is 0: Do not roll over. + max_memory_size = setting('AWS_S3_MAX_MEMORY_SIZE', 0) + + def __init__(self, acl=None, bucket=None, **settings): + # check if some of the settings we've provided as class attributes + # need to be overwritten with values passed in here + for name, value in settings.items(): + if hasattr(self, name): + setattr(self, name, value) + + # For backward-compatibility of old differing parameter names + if acl is not None: + self.default_acl = acl + if bucket is not None: + self.bucket_name = bucket + + self.location = (self.location or '').lstrip('/') + # Backward-compatibility: given the anteriority of the SECURE_URL setting + # we fall back to https if specified in order to avoid the construction + # of unsecure urls. + if self.secure_urls: + self.url_protocol = 'https:' + + self._entries = {} + self._bucket = None + self._connection = None + + if not self.access_key and not self.secret_key: + self.access_key, self.secret_key = self._get_access_keys() + + @property + def connection(self): + if self._connection is None: + self._connection = self.connection_class( + self.access_key, + self.secret_key, + is_secure=self.use_ssl, + calling_format=self.calling_format, + host=self.host, + port=self.port, + ) + return self._connection + + @property + def bucket(self): + """ + Get the current bucket. If there is no current bucket object + create it. + """ + if self._bucket is None: + self._bucket = self._get_or_create_bucket(self.bucket_name) + return self._bucket + + @property + def entries(self): + """ + Get the locally cached files for the bucket. + """ + if self.preload_metadata and not self._entries: + self._entries = dict((self._decode_name(entry.key), entry) + for entry in self.bucket.list(prefix=self.location)) + return self._entries + + def _get_access_keys(self): + """ + Gets the access keys to use when accessing S3. If none + are provided to the class in the constructor or in the + settings then get them from the environment variables. + """ + def lookup_env(names): + for name in names: + value = os.environ.get(name) + if value: + return value + access_key = self.access_key or lookup_env(self.access_key_names) + secret_key = self.secret_key or lookup_env(self.secret_key_names) + return access_key, secret_key + + def _get_or_create_bucket(self, name): + """ + Retrieves a bucket if it exists, otherwise creates it. + """ + try: + return self.connection.get_bucket(name, + validate=self.auto_create_bucket) + except self.connection_response_error: + if self.auto_create_bucket: + bucket = self.connection.create_bucket(name) + bucket.set_acl(self.bucket_acl) + return bucket + raise ImproperlyConfigured("Bucket %s does not exist. Buckets " + "can be automatically created by " + "setting AWS_AUTO_CREATE_BUCKET to " + "``True``." % name) + + def _clean_name(self, name): + """ + Cleans the name so that Windows style paths work + """ + # Normalize Windows style paths + clean_name = posixpath.normpath(name).replace('\\', '/') + + # os.path.normpath() can strip trailing slashes so we implement + # a workaround here. + if name.endswith('/') and not clean_name.endswith('/'): + # Add a trailing slash as it was stripped. + return clean_name + '/' + else: + return clean_name + + def _normalize_name(self, name): + """ + Normalizes the name so that paths like /path/to/ignored/../something.txt + work. We check to make sure that the path pointed to is not outside + the directory specified by the LOCATION setting. + """ + try: + return safe_join(self.location, name) + except ValueError: + raise SuspiciousOperation("Attempted access to '%s' denied." % + name) + + def _encode_name(self, name): + return smart_str(name, encoding=self.file_name_charset) + + def _decode_name(self, name): + return force_unicode(name, encoding=self.file_name_charset) + + def _compress_content(self, content): + """Gzip a given string content.""" + zbuf = StringIO() + zfile = GzipFile(mode='wb', compresslevel=6, fileobj=zbuf) + try: + zfile.write(content.read()) + finally: + zfile.close() + zbuf.seek(0) + content.file = zbuf + content.seek(0) + return content + + def _open(self, name, mode='rb'): + name = self._normalize_name(self._clean_name(name)) + f = self.file_class(name, mode, self) + if not f.key: + raise IOError('File does not exist: %s' % name) + return f + + def _save(self, name, content): + cleaned_name = self._clean_name(name) + name = self._normalize_name(cleaned_name) + headers = self.headers.copy() + content_type = getattr(content, 'content_type', + mimetypes.guess_type(name)[0] or self.key_class.DefaultContentType) + + # setting the content_type in the key object is not enough. + headers.update({'Content-Type': content_type}) + + if self.gzip and content_type in self.gzip_content_types: + content = self._compress_content(content) + headers.update({'Content-Encoding': 'gzip'}) + + content.name = cleaned_name + encoded_name = self._encode_name(name) + key = self.bucket.get_key(encoded_name) + if not key: + key = self.bucket.new_key(encoded_name) + if self.preload_metadata: + self._entries[encoded_name] = key + + key.set_metadata('Content-Type', content_type) + self._save_content(key, content, headers=headers) + return cleaned_name + + def _save_content(self, key, content, headers): + # only pass backwards incompatible arguments if they vary from the default + kwargs = {} + if self.encryption: + kwargs['encrypt_key'] = self.encryption + key.set_contents_from_file(content, headers=headers, + policy=self.default_acl, + reduced_redundancy=self.reduced_redundancy, + rewind=True, **kwargs) + + def delete(self, name): + name = self._normalize_name(self._clean_name(name)) + self.bucket.delete_key(self._encode_name(name)) + + def exists(self, name): + name = self._normalize_name(self._clean_name(name)) + if self.entries: + return name in self.entries + k = self.bucket.new_key(self._encode_name(name)) + return k.exists() + + def listdir(self, name): + name = self._normalize_name(self._clean_name(name)) + # for the bucket.list and logic below name needs to end in / + # But for the root path "" we leave it as an empty string + if name and not name.endswith('/'): + name += '/' + + dirlist = self.bucket.list(self._encode_name(name)) + files = [] + dirs = set() + base_parts = name.split("/")[:-1] + for item in dirlist: + parts = item.name.split("/") + parts = parts[len(base_parts):] + if len(parts) == 1: + # File + files.append(parts[0]) + elif len(parts) > 1: + # Directory + dirs.add(parts[0]) + return list(dirs), files + + def size(self, name): + name = self._normalize_name(self._clean_name(name)) + if self.entries: + entry = self.entries.get(name) + if entry: + return entry.size + return 0 + return self.bucket.get_key(self._encode_name(name)).size + + def modified_time(self, name): + name = self._normalize_name(self._clean_name(name)) + entry = self.entries.get(name) + # only call self.bucket.get_key() if the key is not found + # in the preloaded metadata. + if entry is None: + entry = self.bucket.get_key(self._encode_name(name)) + # Parse the last_modified string to a local datetime object. + return parse_ts_extended(entry.last_modified) + + def url(self, name, headers=None, response_headers=None): + # Preserve the trailing slash after normalizing the path. + name = self._normalize_name(self._clean_name(name)) + if self.custom_domain: + return "%s//%s/%s" % (self.url_protocol, + self.custom_domain, filepath_to_uri(name)) + return self.connection.generate_url(self.querystring_expire, + method='GET', bucket=self.bucket.name, key=self._encode_name(name), + headers=headers, + query_auth=self.querystring_auth, force_http=not self.secure_urls, + response_headers=response_headers) + + def get_available_name(self, name): + """ Overwrite existing file with the same name. """ + if self.file_overwrite: + name = self._clean_name(name) + return name + return super(S3BotoStorage, self).get_available_name(name) diff --git a/storages/backends/sftpstorage.py b/storages/backends/sftpstorage.py new file mode 100644 index 0000000..177a637 --- /dev/null +++ b/storages/backends/sftpstorage.py @@ -0,0 +1,272 @@ +from __future__ import print_function +# SFTP storage backend for Django. +# Author: Brent Tubbs <brent.tubbs@gmail.com> +# License: MIT +# +# Modeled on the FTP storage by Rafal Jonca <jonca.rafal@gmail.com> +# +# Settings: +# +# SFTP_STORAGE_HOST - The hostname where you want the files to be saved. +# +# SFTP_STORAGE_ROOT - The root directory on the remote host into which files +# should be placed. Should work the same way that STATIC_ROOT works for local +# files. Must include a trailing slash. +# +# SFTP_STORAGE_PARAMS (Optional) - A dictionary containing connection +# parameters to be passed as keyword arguments to +# paramiko.SSHClient().connect() (do not include hostname here). See +# http://www.lag.net/paramiko/docs/paramiko.SSHClient-class.html#connect for +# details +# +# SFTP_STORAGE_INTERACTIVE (Optional) - A boolean indicating whether to prompt +# for a password if the connection cannot be made using keys, and there is not +# already a password in SFTP_STORAGE_PARAMS. You can set this to True to +# enable interactive login when running 'manage.py collectstatic', for example. +# +# DO NOT set SFTP_STORAGE_INTERACTIVE to True if you are using this storage +# for files being uploaded to your site by users, because you'll have no way +# to enter the password when they submit the form.. +# +# SFTP_STORAGE_FILE_MODE (Optional) - A bitmask for setting permissions on +# newly-created files. See http://docs.python.org/library/os.html#os.chmod for +# acceptable values. +# +# SFTP_STORAGE_DIR_MODE (Optional) - A bitmask for setting permissions on +# newly-created directories. See +# http://docs.python.org/library/os.html#os.chmod for acceptable values. +# +# Hint: if you start the mode number with a 0 you can express it in octal +# just like you would when doing "chmod 775 myfile" from bash. +# +# SFTP_STORAGE_UID (Optional) - uid of the account that should be set as owner +# of the files on the remote host. You have to be root to set this. +# +# SFTP_STORAGE_GID (Optional) - gid of the group that should be set on the +# files on the remote host. You have to be a member of the group to set this. +# SFTP_KNOWN_HOST_FILE (Optional) - absolute path of know host file, if it isn't +# set "~/.ssh/known_hosts" will be used + + +import getpass +import os +import paramiko +import posixpath +import stat +import urlparse +from datetime import datetime + +from django.conf import settings +from django.core.files.base import File +from django.core.files.storage import Storage + +try: + from cStringIO import StringIO +except ImportError: + from StringIO import StringIO # noqa + + +class SFTPStorage(Storage): + + def __init__(self): + self._host = settings.SFTP_STORAGE_HOST + + # if present, settings.SFTP_STORAGE_PARAMS should be a dict with params + # matching the keyword arguments to paramiko.SSHClient().connect(). So + # you can put username/password there. Or you can omit all that if + # you're using keys. + self._params = getattr(settings, 'SFTP_STORAGE_PARAMS', {}) + self._interactive = getattr(settings, 'SFTP_STORAGE_INTERACTIVE', + False) + self._file_mode = getattr(settings, 'SFTP_STORAGE_FILE_MODE', None) + self._dir_mode = getattr(settings, 'SFTP_STORAGE_DIR_MODE', None) + + self._uid = getattr(settings, 'SFTP_STORAGE_UID', None) + self._gid = getattr(settings, 'SFTP_STORAGE_GID', None) + self._known_host_file = getattr(settings, 'SFTP_KNOWN_HOST_FILE', None) + + self._root_path = settings.SFTP_STORAGE_ROOT + self._base_url = settings.MEDIA_URL + + # for now it's all posix paths. Maybe someday we'll support figuring + # out if the remote host is windows. + self._pathmod = posixpath + + def _connect(self): + self._ssh = paramiko.SSHClient() + + if self._known_host_file is not None: + self._ssh.load_host_keys(self._known_host_file) + else: + # automatically add host keys from current user. + self._ssh.load_host_keys(os.path.expanduser(os.path.join("~", ".ssh", "known_hosts"))) + + # and automatically add new host keys for hosts we haven't seen before. + self._ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + + try: + self._ssh.connect(self._host, **self._params) + except paramiko.AuthenticationException as e: + if self._interactive and 'password' not in self._params: + # If authentication has failed, and we haven't already tried + # username/password, and configuration allows it, then try + # again with username/password. + if 'username' not in self._params: + self._params['username'] = getpass.getuser() + self._params['password'] = getpass.getpass() + self._connect() + else: + raise paramiko.AuthenticationException(e) + except Exception as e: + print(e) + + if not hasattr(self, '_sftp'): + self._sftp = self._ssh.open_sftp() + + @property + def sftp(self): + """Lazy SFTP connection""" + if not hasattr(self, '_sftp'): + self._connect() + return self._sftp + + def _join(self, *args): + # Use the path module for the remote host type to join a path together + return self._pathmod.join(*args) + + def _remote_path(self, name): + return self._join(self._root_path, name) + + def _open(self, name, mode='rb'): + return SFTPStorageFile(name, self, mode) + + def _read(self, name): + remote_path = self._remote_path(name) + return self.sftp.open(remote_path, 'rb') + + def _chown(self, path, uid=None, gid=None): + """Set uid and/or gid for file at path.""" + # Paramiko's chown requires both uid and gid, so look them up first if + # we're only supposed to set one. + if uid is None or gid is None: + attr = self.sftp.stat(path) + uid = uid or attr.st_uid + gid = gid or attr.st_gid + self.sftp.chown(path, uid, gid) + + def _mkdir(self, path): + """Create directory, recursing up to create parent dirs if + necessary.""" + parent = self._pathmod.dirname(path) + if not self.exists(parent): + self._mkdir(parent) + self.sftp.mkdir(path) + + if self._dir_mode is not None: + self.sftp.chmod(path, self._dir_mode) + + if self._uid or self._gid: + self._chown(path, uid=self._uid, gid=self._gid) + + def _save(self, name, content): + """Save file via SFTP.""" + content.open() + path = self._remote_path(name) + dirname = self._pathmod.dirname(path) + if not self.exists(dirname): + self._mkdir(dirname) + + f = self.sftp.open(path, 'wb') + f.write(content.file.read()) + f.close() + + # set file permissions if configured + if self._file_mode is not None: + self.sftp.chmod(path, self._file_mode) + if self._uid or self._gid: + self._chown(path, uid=self._uid, gid=self._gid) + return name + + def delete(self, name): + remote_path = self._remote_path(name) + self.sftp.remove(remote_path) + + def exists(self, name): + # Try to retrieve file info. Return true on success, false on failure. + remote_path = self._remote_path(name) + try: + self.sftp.stat(remote_path) + return True + except IOError: + return False + + def _isdir_attr(self, item): + # Return whether an item in sftp.listdir_attr results is a directory + if item.st_mode is not None: + return stat.S_IFMT(item.st_mode) == stat.S_IFDIR + else: + return False + + def listdir(self, path): + remote_path = self._remote_path(path) + dirs, files = [], [] + for item in self.sftp.listdir_attr(remote_path): + if self._isdir_attr(item): + dirs.append(item.filename) + else: + files.append(item.filename) + return dirs, files + + def size(self, name): + remote_path = self._remote_path(name) + return self.sftp.stat(remote_path).st_size + + def accessed_time(self, name): + remote_path = self._remote_path(name) + utime = self.sftp.stat(remote_path).st_atime + return datetime.fromtimestamp(utime) + + def modified_time(self, name): + remote_path = self._remote_path(name) + utime = self.sftp.stat(remote_path).st_mtime + return datetime.fromtimestamp(utime) + + def url(self, name): + if self._base_url is None: + raise ValueError("This file is not accessible via a URL.") + return urlparse.urljoin(self._base_url, name).replace('\\', '/') + + +class SFTPStorageFile(File): + def __init__(self, name, storage, mode): + self._name = name + self._storage = storage + self._mode = mode + self._is_dirty = False + self.file = StringIO() + self._is_read = False + + @property + def size(self): + if not hasattr(self, '_size'): + self._size = self._storage.size(self._name) + return self._size + + def read(self, num_bytes=None): + if not self._is_read: + self.file = self._storage._read(self._name) + self._is_read = True + + return self.file.read(num_bytes) + + def write(self, content): + if 'w' not in self._mode: + raise AttributeError("File was opened for read-only access.") + self.file = StringIO(content) + self._is_dirty = True + self._is_read = True + + def close(self): + if self._is_dirty: + self._storage._save(self._name, self.file.getvalue()) + self.file.close() diff --git a/storages/backends/symlinkorcopy.py b/storages/backends/symlinkorcopy.py new file mode 100644 index 0000000..14f8e29 --- /dev/null +++ b/storages/backends/symlinkorcopy.py @@ -0,0 +1,62 @@ +import os + +from django.conf import settings +from django.core.files.storage import FileSystemStorage + +__doc__ = """ +I needed to efficiently create a mirror of a directory tree (so that +"origin pull" CDNs can automatically pull files). The trick was that +some files could be modified, and some could be identical to the original. +Of course it doesn't make sense to store the exact same data twice on the +file system. So I created SymlinkOrCopyStorage. + +SymlinkOrCopyStorage allows you to symlink a file when it's identical to +the original file and to copy the file if it's modified. +Of course, it's impossible to know if a file is modified just by looking +at the file, without knowing what the original file was. +That's what the symlinkWithin parameter is for. It accepts one or more paths +(if multiple, they should be concatenated using a colon (:)). +Files that will be saved using SymlinkOrCopyStorage are then checked on their +location: if they are within one of the symlink_within directories, +they will be symlinked, otherwise they will be copied. + +The rationale is that unmodified files will exist in their original location, +e.g. /htdocs/example.com/image.jpg and modified files will be stored in +a temporary directory, e.g. /tmp/image.jpg. +""" + +class SymlinkOrCopyStorage(FileSystemStorage): + """Stores symlinks to files instead of actual files whenever possible + + When a file that's being saved is currently stored in the symlink_within + directory, then symlink the file. Otherwise, copy the file. + """ + def __init__(self, location=settings.MEDIA_ROOT, base_url=settings.MEDIA_URL, + symlink_within=None): + super(SymlinkOrCopyStorage, self).__init__(location, base_url) + self.symlink_within = symlink_within.split(":") + + def _save(self, name, content): + full_path_dst = self.path(name) + + directory = os.path.dirname(full_path_dst) + if not os.path.exists(directory): + os.makedirs(directory) + elif not os.path.isdir(directory): + raise IOError("%s exists and is not a directory." % directory) + + full_path_src = os.path.abspath(content.name) + + symlinked = False + # Only symlink if the current platform supports it. + if getattr(os, "symlink", False): + for path in self.symlink_within: + if full_path_src.startswith(path): + os.symlink(full_path_src, full_path_dst) + symlinked = True + break + + if not symlinked: + super(SymlinkOrCopyStorage, self)._save(name, content) + + return name diff --git a/storages/models.py b/storages/models.py new file mode 100644 index 0000000..78aae72 --- /dev/null +++ b/storages/models.py @@ -0,0 +1,11 @@ +from django.conf import settings + + +if settings.DEFAULT_FILE_STORAGE.endswith('mosso.CloudFilesStorage'): + import warnings + warnings.simplefilter('always', PendingDeprecationWarning) + warnings.warn("The mosso module will be deprecated in version 1.2 of " + "django-storages. The CloudFiles code has been moved into" + "django-cumulus at http://github.com/richleland/django-cumulus.", + PendingDeprecationWarning) + diff --git a/storages/tests/__init__.py b/storages/tests/__init__.py new file mode 100644 index 0000000..e17e6d5 --- /dev/null +++ b/storages/tests/__init__.py @@ -0,0 +1,2 @@ +from storages.tests.hashpath import * +from storages.tests.s3boto import * diff --git a/storages/tests/hashpath.py b/storages/tests/hashpath.py new file mode 100644 index 0000000..5cc4d65 --- /dev/null +++ b/storages/tests/hashpath.py @@ -0,0 +1,35 @@ +import os +import shutil + +from django.test import TestCase +from django.core.files.base import ContentFile +from django.conf import settings + +from storages.backends.hashpath import HashPathStorage + +TEST_PATH_PREFIX = 'django-storages-test' + + +class HashPathStorageTest(TestCase): + + def setUp(self): + self.test_path = os.path.join(settings.MEDIA_ROOT, TEST_PATH_PREFIX) + self.storage = HashPathStorage(location=self.test_path) + + # make sure the profile upload folder exists + if not os.path.exists(self.test_path): + os.makedirs(self.test_path) + + def tearDown(self): + # remove uploaded profile picture + if os.path.exists(self.test_path): + shutil.rmtree(self.test_path) + + def test_save_same_file(self): + """ + saves a file twice, the file should only be stored once, because the + content/hash is the same + """ + path_1 = self.storage.save('test', ContentFile('new content')) + path_2 = self.storage.save('test', ContentFile('new content')) + self.assertEqual(path_1, path_2) diff --git a/storages/tests/s3boto.py b/storages/tests/s3boto.py new file mode 100644 index 0000000..3f49503 --- /dev/null +++ b/storages/tests/s3boto.py @@ -0,0 +1,308 @@ +import mock
+import datetime
+import urlparse
+
+from django.test import TestCase
+from django.core.files.base import ContentFile
+
+from boto.s3.key import Key
+
+from storages.backends import s3boto
+
+__all__ = (
+ 'ParseTsExtendedCase',
+ 'SafeJoinTest',
+ 'S3BotoStorageTests',
+ #'S3BotoStorageFileTests',
+)
+
+class ParseTsExtendedCase(TestCase):
+ def test_normal(self):
+ value = s3boto.parse_ts_extended("Wed, 13 Mar 2013 12:45:49 GMT")
+ self.assertEquals(value, datetime.datetime(2013, 3, 13, 12, 45, 49))
+
+class S3BotoTestCase(TestCase):
+ @mock.patch('storages.backends.s3boto.S3Connection')
+ def setUp(self, S3Connection):
+ self.storage = s3boto.S3BotoStorage()
+ self.storage._connection = mock.MagicMock()
+
+class SafeJoinTest(TestCase):
+ def test_normal(self):
+ path = s3boto.safe_join("", "path/to/somewhere", "other", "path/to/somewhere")
+ self.assertEquals(path, "path/to/somewhere/other/path/to/somewhere")
+
+ def test_with_dot(self):
+ path = s3boto.safe_join("", "path/./somewhere/../other", "..",
+ ".", "to/./somewhere")
+ self.assertEquals(path, "path/to/somewhere")
+
+ def test_base_url(self):
+ path = s3boto.safe_join("base_url", "path/to/somewhere")
+ self.assertEquals(path, "base_url/path/to/somewhere")
+
+ def test_base_url_with_slash(self):
+ path = s3boto.safe_join("base_url/", "path/to/somewhere")
+ self.assertEquals(path, "base_url/path/to/somewhere")
+
+ def test_suspicious_operation(self):
+ self.assertRaises(ValueError,
+ s3boto.safe_join, "base", "../../../../../../../etc/passwd")
+
+ def test_trailing_slash(self):
+ """
+ Test safe_join with paths that end with a trailing slash.
+ """
+ path = s3boto.safe_join("base_url/", "path/to/somewhere/")
+ self.assertEquals(path, "base_url/path/to/somewhere/")
+
+ def test_trailing_slash_multi(self):
+ """
+ Test safe_join with multiple paths that end with a trailing slash.
+ """
+ path = s3boto.safe_join("base_url/", "path/to/" "somewhere/")
+ self.assertEquals(path, "base_url/path/to/somewhere/")
+
+
+class S3BotoStorageTests(S3BotoTestCase):
+
+ def test_clean_name(self):
+ """
+ Test the base case of _clean_name
+ """
+ path = self.storage._clean_name("path/to/somewhere")
+ self.assertEqual(path, "path/to/somewhere")
+
+ def test_clean_name_normalize(self):
+ """
+ Test the normalization of _clean_name
+ """
+ path = self.storage._clean_name("path/to/../somewhere")
+ self.assertEqual(path, "path/somewhere")
+
+ def test_clean_name_trailing_slash(self):
+ """
+ Test the _clean_name when the path has a trailing slash
+ """
+ path = self.storage._clean_name("path/to/somewhere/")
+ self.assertEqual(path, "path/to/somewhere/")
+
+ def test_clean_name_windows(self):
+ """
+ Test the _clean_name when the path has a trailing slash
+ """
+ path = self.storage._clean_name("path\\to\\somewhere")
+ self.assertEqual(path, "path/to/somewhere")
+
+ def test_storage_url(self):
+ """
+ Test URL generation.
+ """
+ self.storage.custom_domain = 'example.com'
+
+ # We expect no leading slashes in the path,
+ # and trailing slashes should be preserved.
+ self.assertEqual(self.storage.url(''), 'https://example.com/')
+ self.assertEqual(self.storage.url('path'), 'https://example.com/path')
+ self.assertEqual(self.storage.url('path/'), 'https://example.com/path/')
+ self.assertEqual(self.storage.url('path/1'), 'https://example.com/path/1')
+ self.assertEqual(self.storage.url('path/1/'), 'https://example.com/path/1/')
+
+ def test_storage_save(self):
+ """
+ Test saving a file
+ """
+ name = 'test_storage_save.txt'
+ content = ContentFile('new content')
+ self.storage.save(name, content)
+ self.storage.bucket.get_key.assert_called_once_with(name)
+
+ key = self.storage.bucket.get_key.return_value
+ key.set_metadata.assert_called_with('Content-Type', 'text/plain')
+ key.set_contents_from_file.assert_called_with(
+ content,
+ headers={'Content-Type': 'text/plain'},
+ policy=self.storage.default_acl,
+ reduced_redundancy=self.storage.reduced_redundancy,
+ rewind=True
+ )
+
+ def test_storage_save_gzip(self):
+ """
+ Test saving a file with gzip enabled.
+ """
+ if not s3boto.S3BotoStorage.gzip: # Gzip not available.
+ return
+ name = 'test_storage_save.css'
+ content = ContentFile("I should be gzip'd")
+ self.storage.save(name, content)
+ key = self.storage.bucket.get_key.return_value
+ key.set_metadata.assert_called_with('Content-Type', 'text/css')
+ key.set_contents_from_file.assert_called_with(
+ content,
+ headers={'Content-Type': 'text/css', 'Content-Encoding': 'gzip'},
+ policy=self.storage.default_acl,
+ reduced_redundancy=self.storage.reduced_redundancy,
+ rewind=True,
+ )
+
+ def test_compress_content_len(self):
+ """
+ Test that file returned by _compress_content() is readable.
+ """
+ if not s3boto.S3BotoStorage.gzip: # Gzip not available.
+ return
+ content = ContentFile("I should be gzip'd")
+ content = self.storage._compress_content(content)
+ self.assertTrue(len(content.read()) > 0)
+
+ def test_storage_open_write(self):
+ """
+ Test opening a file in write mode
+ """
+ name = 'test_open_for_writing.txt'
+ content = 'new content'
+
+ # Set the ACL header used when creating/writing data.
+ self.storage.bucket.connection.provider.acl_header = 'x-amz-acl'
+ # Set the mocked key's bucket
+ self.storage.bucket.get_key.return_value.bucket = self.storage.bucket
+ # Set the name of the mock object
+ self.storage.bucket.get_key.return_value.name = name
+
+ file = self.storage.open(name, 'w')
+ self.storage.bucket.get_key.assert_called_with(name)
+
+ file.write(content)
+ self.storage.bucket.initiate_multipart_upload.assert_called_with(
+ name,
+ headers={
+ 'Content-Type': 'text/plain',
+ 'x-amz-acl': 'public-read',
+ },
+ reduced_redundancy=self.storage.reduced_redundancy,
+ )
+
+ # Save the internal file before closing
+ _file = file.file
+ file.close()
+ file._multipart.upload_part_from_file.assert_called_with(
+ _file, 1, headers=self.storage.headers,
+ )
+ file._multipart.complete_upload.assert_called_once()
+
+ def test_storage_exists(self):
+ key = self.storage.bucket.new_key.return_value
+ key.exists.return_value = True
+ self.assertTrue(self.storage.exists("file.txt"))
+
+ def test_storage_exists_false(self):
+ key = self.storage.bucket.new_key.return_value
+ key.exists.return_value = False
+ self.assertFalse(self.storage.exists("file.txt"))
+
+ def test_storage_delete(self):
+ self.storage.delete("path/to/file.txt")
+ self.storage.bucket.delete_key.assert_called_with("path/to/file.txt")
+
+ def test_storage_listdir_base(self):
+ file_names = ["some/path/1.txt", "2.txt", "other/path/3.txt", "4.txt"]
+
+ self.storage.bucket.list.return_value = []
+ for p in file_names:
+ key = mock.MagicMock(spec=Key)
+ key.name = p
+ self.storage.bucket.list.return_value.append(key)
+
+ dirs, files = self.storage.listdir("")
+
+ self.assertEqual(len(dirs), 2)
+ for directory in ["some", "other"]:
+ self.assertTrue(directory in dirs,
+ """ "%s" not in directory list "%s".""" % (
+ directory, dirs))
+
+ self.assertEqual(len(files), 2)
+ for filename in ["2.txt", "4.txt"]:
+ self.assertTrue(filename in files,
+ """ "%s" not in file list "%s".""" % (
+ filename, files))
+
+ def test_storage_listdir_subdir(self):
+ file_names = ["some/path/1.txt", "some/2.txt"]
+
+ self.storage.bucket.list.return_value = []
+ for p in file_names:
+ key = mock.MagicMock(spec=Key)
+ key.name = p
+ self.storage.bucket.list.return_value.append(key)
+
+ dirs, files = self.storage.listdir("some/")
+ self.assertEqual(len(dirs), 1)
+ self.assertTrue('path' in dirs,
+ """ "path" not in directory list "%s".""" % (dirs,))
+
+ self.assertEqual(len(files), 1)
+ self.assertTrue('2.txt' in files,
+ """ "2.txt" not in files list "%s".""" % (files,))
+
+ def test_storage_size(self):
+ key = self.storage.bucket.get_key.return_value
+ key.size = 4098
+
+ name = 'file.txt'
+ self.assertEqual(self.storage.size(name), key.size)
+
+ def test_storage_url(self):
+ name = 'test_storage_size.txt'
+ url = 'http://aws.amazon.com/%s' % name
+ self.storage.connection.generate_url.return_value = url
+
+ self.assertEquals(self.storage.url(name), url)
+ self.storage.connection.generate_url.assert_called_with(
+ self.storage.querystring_expire,
+ method='GET',
+ bucket=self.storage.bucket.name,
+ key=name,
+ query_auth=self.storage.querystring_auth,
+ force_http=not self.storage.secure_urls,
+ headers=None,
+ response_headers=None,
+ )
+
+ def test_generated_url_is_encoded(self):
+ self.storage.custom_domain = "mock.cloudfront.net"
+ filename = "whacky & filename.mp4"
+ url = self.storage.url(filename)
+ parsed_url = urlparse.urlparse(url)
+ self.assertEqual(parsed_url.path,
+ "/whacky%20%26%20filename.mp4")
+
+#class S3BotoStorageFileTests(S3BotoTestCase):
+# def test_multipart_upload(self):
+# nparts = 2
+# name = self.prefix_path("test_multipart_upload.txt")
+# mode = 'w'
+# f = s3boto.S3BotoStorageFile(name, mode, self.storage)
+# content_length = 1024 * 1024# 1 MB
+# content = 'a' * content_length
+#
+# bytes = 0
+# target = f._write_buffer_size * nparts
+# while bytes < target:
+# f.write(content)
+# bytes += content_length
+#
+# # make the buffer roll over so f._write_counter
+# # is incremented
+# f.write("finished")
+#
+# # verify upload was multipart and correctly partitioned
+# self.assertEqual(f._write_counter, nparts)
+#
+# # complete the upload
+# f.close()
+#
+# # verify that the remaining buffered bytes were
+# # uploaded when the file was closed.
+# self.assertEqual(f._write_counter, nparts+1)
diff --git a/storages/utils.py b/storages/utils.py new file mode 100644 index 0000000..810e0c5 --- /dev/null +++ b/storages/utils.py @@ -0,0 +1,9 @@ +from django.conf import settings + + +def setting(name, default=None): + """ + Helper function to get a Django setting by name or (optionally) return + a default (or else ``None``). + """ + return getattr(settings, name, default) diff --git a/tests.py b/tests.py new file mode 100644 index 0000000..98c5536 --- /dev/null +++ b/tests.py @@ -0,0 +1,63 @@ +import os +import sys +import django + +BASE_PATH = os.path.dirname(__file__) + +def main(): + """ + Standalone django model test with a 'memory-only-django-installation'. + You can play with a django model without a complete django app installation. + http://www.djangosnippets.org/snippets/1044/ + """ + sys.exc_clear() + + os.environ["DJANGO_SETTINGS_MODULE"] = "django.conf.global_settings" + from django.conf import global_settings + + global_settings.INSTALLED_APPS = ( + 'django.contrib.auth', + 'django.contrib.sessions', + 'django.contrib.contenttypes', + 'storages', + ) + if django.VERSION > (1,2): + global_settings.DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.sqlite3', + 'NAME': os.path.join(BASE_PATH, 'connpass.sqlite'), + 'USER': '', + 'PASSWORD': '', + 'HOST': '', + 'PORT': '', + } + } + else: + global_settings.DATABASE_ENGINE = "sqlite3" + global_settings.DATABASE_NAME = ":memory:" + + global_settings.ROOT_URLCONF='beproud.django.authutils.tests.test_urls' + global_settings.MIDDLEWARE_CLASSES = ( + 'django.middleware.common.CommonMiddleware', + 'django.contrib.sessions.middleware.SessionMiddleware', + 'django.middleware.csrf.CsrfViewMiddleware', + 'django.contrib.auth.middleware.AuthenticationMiddleware', + 'django.contrib.messages.middleware.MessageMiddleware', + 'beproud.django.authutils.middleware.AuthMiddleware', + ) + global_settings.DEFAULT_FILE_STORAGE = 'backends.s3boto.S3BotoStorage' + global_settings.AWS_IS_GZIPPED = True + global_settings.SECRET_KEY = "tralala" + + from django.test.utils import get_runner + test_runner = get_runner(global_settings) + + if django.VERSION > (1,2): + test_runner = test_runner() + failures = test_runner.run_tests(['storages']) + else: + failures = test_runner(['storages'], verbosity=1) + sys.exit(failures) + +if __name__ == '__main__': + main() @@ -0,0 +1,88 @@ +# content of: tox.ini , put in same dir as setup.py +[tox] +envlist = django12-py26,django12-py27, + django13-py26,django13-py27, + django14-py26,django14-py27, + django15-py26,django15-py27 + +[base] +deps = + mock==1.0.1 + boto==2.7.0 + +[testenv] +commands=python setup.py test + +[testenv:dev] +deps= + mock==1.0.1 + -egit://github.com/boto/boto.git@develop#egg=boto + -egit://github.com/django/django.git#egg=django + +##### Django 1.2 #### + +[django12] +deps = + django==1.2.7 + {[base]deps} + +[testenv:django12-py26] +basepython=python2.6 +deps= + {[django12]deps} + +[testenv:django12-py27] +basepython=python2.7 +deps= + {[django12]deps} + +##### Django 1.3 #### + +[django13] +deps = + django==1.3.5 + {[base]deps} + +[testenv:django13-py26] +basepython=python2.6 +deps= + {[django13]deps} + +[testenv:django13-py27] +basepython=python2.7 +deps= + {[django13]deps} + +##### Django 1.4 #### + +[django14] +deps = + django==1.4.3 + {[base]deps} + +[testenv:django14-py26] +basepython=python2.6 +deps= + {[django14]deps} + +[testenv:django14-py27] +basepython=python2.7 +deps= + {[django14]deps} + +##### Django 1.5 #### + +[django15] +deps = + django==1.5.1 + {[base]deps} + +[testenv:django15-py26] +basepython=python2.6 +deps= + {[django15]deps} + +[testenv:django15-py27] +basepython=python2.7 +deps= + {[django15]deps} |