Generating an InlineModelAdmin Form on the fly in Django

I'm adding drag/drop uploading to the django admin for one of our open source projects called Stager. A blog post about that will follow, it's not screen-shot ready yet.  While doing this I knew we needed a pretty seamless transition after the upload finished, and that we would have to refresh the inline.  I didn't want a full page refresh, so let's ajax it in.

For these examples just assume that we have a parent CompAdmin which has an model of Comp and an inline called CompSlideInline.  We store the instance of the Comp in comp.

from django.template import loader, Context
from django.contrib.admin import helpers
from django.db import transaction
from django.contrib import admin

comp = Comp.objects.get(id=comp_id)
#get the current site
admin_site = admin.site
compAdmin = CompAdmin(Comp, admin_site)

#get all possible inlines for the parent Admin
inline_instances = compAdmin.get_inline_instances(request)
prefixes = {}

for FormSet, inline in zip(compAdmin.get_formsets(request, comp), inline_instances):
    #get the inline of interest and generate it's formset
    if isinstance(inline, CompSlideInline):
        prefix = FormSet.get_default_prefix()
        prefixes[prefix] = prefixes.get(prefix, 0) + 1
        if prefixes[prefix] != 1 or not prefix:
            prefix = "%s-%s" % (prefix, prefixes[prefix])
        formset = FormSet(instance=comp, prefix=prefix, queryset=inline.queryset(request))

#get possible fieldsets, readonly, and prepopulated information for the parent Admin
fieldsets = list(inline.get_fieldsets(request, comp))
readonly = list(inline.get_readonly_fields(request, comp))
prepopulated = dict(inline.get_prepopulated_fields(request, comp))

#generate the inline formset
inline_admin_formset = helpers.InlineAdminFormSet(inline, formset,
            fieldsets, prepopulated, readonly, model_admin=compAdmin)

#render the template
t = loader.get_template('tabular.html')
c = Context({ 'inline_admin_formset': inline_admin_formset })
rendered = t.render(c)

Getting Started with Solr and Django

Solr is a very powerful search tool and it is pretty easy to get the basics, such as full text search, facets, and related assets up and running pretty quickly. We will be using haystack to do the communication between Django and Solr. All code for this can be viewed on github.

Install

Assuming you already have Django up and running, the first thing we need to do is install Solr.

curl -O http://mirrors.gigenet.com/apache/lucene/solr/4.0.0-BETA/apache-solr-4.0.0-BETA.zip
unzip apache-solr-4.0.0-BETA.zip
cd apache-solr-4.0.0-BETA
cd example
java -jar start.jar

Next install pysolr and haystack. (At the time of this writing the git checkout of haystack works better with the Solr 4.0 beta then the 1.2.7 that's in pip.)

pip install pysolr
pip install -e https://github.com/toastdriven/django-haystack.git

Add 'haystack' to INSTALLED_APPS in settings.py and add the following haystack connection:

HAYSTACK_CONNECTIONS = { 
    'default': { 
        'ENGINE': 'haystack.backends.solr_backend.SolrEngine', 
        'URL': 'http://127.0.0.1:8983/solr' 
    }, 
}

For the example, we're going to create a simple job database that a recruiter might use. Here is the model:

from django.db import models
from django.contrib.localflavor.us
import models as us_models

JOB_TYPES = (
    ('pt', 'Part Time'),
    ('ft', 'Full Time'),
    ('ct', 'Contract')
)

class Company(models.Model):
    name = models.CharField(max_length=64)
    address = models.TextField(blank=True, null=True)
    contact_email = models.EmailField()

    def __unicode__(self):
        return self.name

class Location(models.Model):
    city = models.CharField(max_length=64)
    state = us_models.USStateField()

    def __unicode__(self):
        return "%s, %s" % (self.city, self.state)

class Job(models.Model):
    name = models.CharField(max_length=64)
    description = models.TextField()
    salary = models.CharField(max_length=64, blank=True, null=True)
    type = models.CharField(max_length=2, choices=JOB_TYPES)
    company = models.ForeignKey(Company, related_name='jobs')
    location = models.ForeignKey(Location, related_name='location_jobs')
    contact_email = models.EmailField(blank=True, null=True)
    added_at = models.DateTimeField(auto_now=True)

    def __unicode__(self):
        return self.name

    def get_contact_email(self):
        if self.contact_email:
            return self.contact_email

        return self.company.contact_email

The next step is to create the SearchIndex object that will be used to transpose to data to Solr. save this as search_indexes.py in the same folder as your models.py. The text field with its template will be used for full text search on Solr. The other two fields will be used to faceted (drill down) navigation. For more details on this file, check out the [haystack tutorial](http://django- haystack.readthedocs.org/en/latest/tutorial.html#handling-data).

class JobIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, use_template=True)
    type = indexes.CharField(model_attr='type', faceted=True)
    location = indexes.CharField(model_attr='location', faceted=True)

    def get_model(self):
        return Job

    def index_queryset(self):
        return self.get_model().objects.all()

Create the search index template in your template folder with the following naming convention: search/indexes/[app]/[model]_text.txt

For us, this is templates/search/indexes/jobs/job_text.txt


{{ object.name }}
{{ object.description }}
{{ object.salary }}
{{ object.type }}
{{ object.added_at }}

Now, lets get our data into Solr. Run ./manage.py build_solr_schema to generate a schema.xml file. Move this into example\solr\conf in your Solr install. Note: if using Solr 4, edit this file and replace stopwords_en.txt with lang/stopwords_en.txt in all locations. To test everything and load your data, run: manage.py rebuild_index Subsequent updates can be made with: manage.py update_index.

If that all worked we can start working on the front-end to see the data in Django. Add this to your urls.py

(r'^$', include('haystack.urls')),

At this point there are at least two templates we'll need. One for the search results page, and a sub-template to represent each item we are pulling back. My example uses twitter bootstrap for some layout help and styling, see my base.html [here](https://github.com/broderboy/django-solr- demo/blob/master/templates/base.html) if interested.

Create templates/search/search.html This gives you a basic search form, the results, and pagination for a number of results


{% extends 'base.html' %}

{% block hero_text %}Search{% endblock %}
{% block header %}<p>Click around!</p>{% endblock %}


{% block content %}
<div class="span12">
    <h1>Search</h1>
    <form method="get" action="." class=".form-search">
        <table>
            {{ form.as_table }}
        </table>
        <input type="submit" value="Search">
    </form>
</div>
        {% if query %}
            <div class="span2">
                <h3>Filter</h3>
                {% if facets.fields.type %}
                    <div>
                        <h4>Type</h4>
                        <ul>
                        {% for type in facets.fields.type %}
                            <li><a href="{{ request.get_full_path }}&amp;selected_facets=type_exact:{{ type.0|urlencode }}">{{ type.0 }}</a> ({{ type.1 }})</li>
                        {% endfor %}
                        </ul>
                    </div>
                {% endif %}
                {% if facets.fields.location %}
                    <div>
                        <h4>Location</h4>
                        <ul>
                        {% for location in facets.fields.location %}
                            <li><a href="{{ request.get_full_path }}&amp;selected_facets=location_exact:{{ location.0|urlencode }}">{{ location.0 }}</a> ({{ location.1 }})</li>
                        {% endfor %}
                        </ul>
                    </div>
                {% endif %}
            </div>
            <div class="span6">
                <h3>Results</h3>
                <div class="accordion" id="accordion2">
                    {% for result in page.object_list %}
                        {% include 'search/_result_object.html' %}
                    {% empty %}
                        <p>No results found.</p>
                    {% endfor %}
                </div>

                {% if page.has_previous or page.has_next %}
                    <div>
                        {% if page.has_previous %}<a href="?q={{ query }}&amp;page={{ page.previous_page_number }}">{% endif %}&laquo; Previous{% if page.has_previous %}</a>{% endif %}
                        |
                        {% if page.has_next %}<a href="?q={{ query }}&amp;page={{ page.next_page_number }}">{% endif %}Next &raquo;{% if page.has_next %}</a>{% endif %}
                    </div>
                {% endif %}
            </div>
        {% else %}
            <div class="span6">
                {# Show some example queries to run, maybe query syntax, something else? #}
            </div>
        {% endif %}
{% endblock %}

And the templates/search/_result_object.txt


{% load more_like_this %}


{% with obj=result.object %}
<div class="accordion-group">
    <div class="accordion-heading">
        <a class="accordion-toggle" data-toggle="collapse" data-parent="#accordion2" href="#collapse_{{ obj.id }}">
            {{ obj.name }}
        </a>
        <div style="padding: 8px 15px;">
            <p>Company: {{ obj.company }}</p>
            <p>Type: {{ obj.type }}</p>
            {% if obj.salary %}<p>Salary: {{ obj.salary }}</p>{% endif %}
            <p>Location: {{ obj.location }}</p>
        </div>
    </div>
    <div id="collapse_{{ obj.id }}" class="accordion-body collapse in">
        <div class="accordion-inner">
            <p>Contact: <a href="mailto:{{ obj.get_contact_email }}">{{ obj.get_contact_email }}</a></p>
            {{ obj.description }}
            {% more_like_this obj as related_content limit 5  %}
            {% if related_content %}
                <div>
                    <br>
                    <p><strong>Related:</strong></p>
                    <ul>
                        {% for related in related_content %}
                            <li><a>{{ related.object.name }}</a></li>
                        {% endfor %}
                    </ul>
                </div>
            {% endif %}
        </div>
    </div>
</div>
{% endwith %}

Start up your dev server for search!

Adding Related Items is as simple as using the related_content tag in the haystack more_like_this tag library and tweaking out Solr config. Open up solrconfig.xml and add a MoreLikeThisHandler within thetag:

<requesthandler name="/mlt" class="solr.MoreLikeThisHandler"></requesthandler>

Our full _result_object.html now looks like this:


{% load more_like_this %}


{% with obj=result.object %}
<div class="accordion-group">
    <div class="accordion-heading">
        <a class="accordion-toggle" data-toggle="collapse" data-parent="#accordion2" href="#collapse_{{ obj.id }}">
            {{ obj.name }}
        </a>
        <div style="padding: 8px 15px;">
            <p>Company: {{ obj.company }}</p>
            <p>Type: {{ obj.type }}</p>
            {% if obj.salary %}<p>Salary: {{ obj.salary }}</p>{% endif %}
            <p>Location: {{ obj.location }}</p>
        </div>
    </div>
    <div id="collapse_{{ obj.id }}" class="accordion-body collapse in">
        <div class="accordion-inner">
            <p>Contact: <a href="mailto:{{ obj.get_contact_email }}">{{ obj.get_contact_email }}</a></p>
            {{ obj.description }}
            {% more_like_this obj as related_content limit 5  %}
            {% if related_content %}
                <div>
                    <br />
                    <p><strong>Related:</strong></p>
                    <ul>
                        {% for related in related_content %}
                            <li><a>{{ related.object.name }}</a></li>
                        {% endfor %}
                    </ul>
                </div>
            {% endif %}
        </div>
    </div>
</div>
{% endwith %}

Facets

To get our type and location facets, we'll have to add them to a queryset and pass this to a FacetedSearchView instead of the default one. urls.py now looks like this:

from django.conf.urls import patterns, include, url
from django.contrib import admin
admin.autodiscover()
from haystack.forms import FacetedSearchForm
from haystack.query import SearchQuerySet
from haystack.views import FacetedSearchView

sqs = SearchQuerySet().facet('type').facet('location')

urlpatterns = patterns('haystack.views',
    url(r'^$', FacetedSearchView(form_class=FacetedSearchForm, searchqueryset=sqs), name='haystack_search'),
)

urlpatterns = urlpatterns + patterns('',
    url(r'^admin/', include(admin.site.urls)),
    #(r'^', include('haystack.urls')),
)
- See more at: http://timbroder.com/2012/08/getting-started-with-solr-and-django.html?preview=true&preview_id=1167&preview_nonce=f6c209628d#sthash.2NVpbeDJ.dpuf

Then, we can use the generated facets in the search template in the facets variable


{% extends 'base.html' %}

{% block hero_text %}Search{% endblock %}
{% block header %}</p><p>Click around!</p>{% endblock %}


{% block content %}
<div class="span12">
    <h1>Search</h1>
    <form method="get" action="." class=".form-search">
        <table>
            {{ form.as_table }}
        </table>
        <input type="submit" value="Search"/>
    </form>
</div>
        {% if query %}
            <div class="span2">
                <h3>Filter</h3>
                {% if facets.fields.type %}
                    <div>
                        <h4>Type</h4>
                        <ul>
                        {% for type in facets.fields.type %}
                            <li><a href="{{ request.get_full_path }}&amp;selected_facets=type_exact:{{ type.0|urlencode }}">{{ type.0 }}</a> ({{ type.1 }})</li>
                        {% endfor %}
                        </ul>
                    </div>
                {% endif %}
                {% if facets.fields.location %}
                    <div>
                        <h4>Location</h4>
                        <ul>
                        {% for location in facets.fields.location %}
                            <li><a href="{{ request.get_full_path }}&amp;selected_facets=location_exact:{{ location.0|urlencode }}">{{ location.0 }}</a> ({{ location.1 }})</li>
                        {% endfor %}
                        </ul>
                    </div>
                {% endif %}
            </div>
            <div class="span6">
                <h3>Results</h3>
                <div class="accordion" id="accordion2">
                    {% for result in page.object_list %}
                        {% include 'search/_result_object.html' %}
                    {% empty %}
                        <p>No results found.</p>
                    {% endfor %}
                </div>

                {% if page.has_previous or page.has_next %}
                    <div>
                        {% if page.has_previous %}<a href="?q={{ query }}&amp;page={{ page.previous_page_number }}">{% endif %}&laquo; Previous{% if page.has_previous %}</a>{% endif %}
                        |
                        {% if page.has_next %}<a href="?q={{ query }}&amp;page={{ page.next_page_number }}">{% endif %}Next &raquo;{% if page.has_next %}</a>{% endif %}
                    </div>
                {% endif %}
            </div>
        {% else %}
            <div class="span6">
                {# Show some example queries to run, maybe query syntax, something else? #}
            </div>
        {% endif %}
{% endblock %}

And we're done! As I said, check out the [haystack documentation](http ://django-haystack.readthedocs.org/en/latest/index.html) for more information. Leave any questions in the comments and I'll be sure to answer them

[![](http://timbroder.com/wp- content/uploads/2012/08/solr2.png)](http://timbroder.com/wp- content/uploads/2012/08/solr2.png)

Hiding fields in the Django admin based on permission

Quick and easy if you have one group of users that can only edit an asset, and another group that has publishing rights

def remove_from_fieldsets(fieldsets, fields):
    for fieldset in fieldsets:
        for field in fields:
            if field in fieldset[1]['fields']:
                new_fields = []
                for new_field in fieldset[1]['fields']:
                    if not new_field in fields:
                        new_fields.append(new_field)

                fieldset[1]['fields'] = tuple(new_fields)
                break

class PositionAdmin(admin.ModelAdmin):
    ...
    def get_fieldsets(self, request, obj=None):
        fieldsets = super(PositionAdmin, self).get_fieldsets(request, obj)

        if not request.user.is_superuser and request.user.groups.filter(name='publisher').count() == 0:
            remove_from_fieldsets(fieldsets, ('is_published',))
        return fieldsets

Making virtualenv on windows with powershell a little cleaner

While I code on a mac at home, I can't live without my giant dual screens and solid state drive at work so I'm on a windows 7 box.  Most of the time it's fine, does everything I need, and I'm happy.  I became full of rage for the first time last week trying to properly get virtualenv to play nice with powershell.  (If you code on windows and are in the terminal a lot, switch to powershell, its great and comes with windows 7. There is a download for Windows XP) I'm not going to recap how to set up virtualenv for your project as there is a great walk through on that here.  The issue on windows is around when you want to activate your project.  Powershell has a restricted execution policy turned on by default. The manual way around this is to run powershell as an administrator, and run this: [code] Set-ExecutionPolicy Unrestricted [/code] Works, but that's an extra click.  You can also change this value permanently in the registry at the key listed below, but that didn't seem to stick when opening powershell through launchy [code] HKLM\Software\Microsoft\PowerShell\1\ShellIds\Microsoft.PowerShell [/code] Enter my hacked up solution. Create a shortcut for powershell with these parameters:

Target: %SystemRoot%\syswow64\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy
Unrestricted Start In: %HOMEDRIVE%%HOMEPATH% 

Then, if your workspace and projects are set up relatively the same, you can create a powershell script (or a cmd script if not using powershell), named workon.ps1 that looks something like this:

$ENV:PYTHONPATH="" 
cd C:\Users\tbroder\workspace\$args\ .\myenv\Scripts\activate

I threw this in my C:\Python26\Scripts folder.  It assumes your project lives in a workspace folder, that your project name is a single word, and that all of your virtualenvs are called myenv.  Example of using it below:

Windows PowerShell Copyright (C) 2009 Microsoft Corporation. All rights reserved. PS
C:\Users\tbroder&gt; workon gsb (myenv) PS C:\Users\tbroder\workspace\gsb&gt;

Developing with multiple versions of Django on windows

At work, we have sites that use various versions of Django, so there is a need to switch packages.  On my mac I use virtualenv to handle this, but I've never quite gotten it to play nice with windows, and just switching Django versions has sufficed so far. If you can set up virtualenv, that's a much better solution

  1. Download junction (think symbolic links for windows). Put the executable somewhere in your path. I put it in my Python scripts folder (C:\Python26\Scripts)
  2. Plan out your directory structure: Make sure you don't have any stock django eggs or folders in your site-packages folder
  3. Create a django_veresions folder in site-packages (C:\Python26\Lib\site-packages\django_versions)
  4. In this folder I put my different django installs C:\Python26\Lib\site-packages\django_versions\1.1\django C:\Python26\Lib\site-packages\django_versions\1.3\django C:\Python26\Lib\site-packages\django_versions\1.4\django
  5. I also put a txt file inside the django folder to easily see what version is there (this will be helpful later to double check) ```C:\Python26\Lib\site-packages\django_versions\1.4\django\1.4.txt
  6. Create a file called djangoversion.cmd and drop it in C:\Python26\Scripts junction -d C:\Python26\Lib\site-packages\django junction C:\Python26\Lib\site-packages\django C:\Python26\Lib\site-packages\django_versions\%1\django
  7. The first line removes any links that may have been there. The second line creates a new link to the version you'll pass in.
  8. Now, to switch versions, just run this from the command line: djangoversion 1.4 PS C:\Users\tbroder> djangoversion 1.4 C:\Users\tbroder>junction -d C:\Python26\Lib\site-packages\django Junction v1.06 - Windows junction creator and reparse point viewer Copyright (C) 2000-2010 Mark Russinovich Sysinternals - www.sysinternals.com Deleted C:\Python26\Lib\site-packages\django. C:\Users\tbroder>junction C:\Python26\Lib\site-packages\django C:\Python26\Lib\site-packages\django_versions\1.4\django Junction v1.06 - Windows junction creator and reparse point viewer Copyright (C) 2000-2010 Mark Russinovich Sysinternals - www.sysinternals.com Created: C:\Python26\Lib\site-packages\django Targetted at: C:\Python26\Lib\site-packages\django_versions\1.4\django PS C:\Users\tbroder>
  9. Switch as needed

Some reading from django-nyc

Went to the django-nyc meetup last night, it was in Manhattan and much easier to get to then when it's at huge in bk.  Wrote down a few things to read up on:

select foo, count(*) from bar group by foo in django

Every once in a while you need some old fashion SQL style queries in django. This is a common one for reporting and aggregation.  Its fairly easy to replicate in a queryset.  Say I wanted to get the authors and the number of articles they have written going back to the beginning of 2009 to the present:

from django.db.models import Count
Article.objects
    .filter(created_date__gte=datetime.datetime(2009,1,1))
    .values('author')
    .annotate(Count('author'))

The result:

[{'author__count': 1028, 'author': 17L}, {'author__count': 9, 'author': 9L}, {'author__count': 39, 'author': 12L}, {'author__count': 581, 'author': 10L}, {'author__count': 15, 'author': 7L}, {'author__count': 366, 'author': 13L}, {'author__count': 233, 'author': 5L}, {'author__count': 167, 'author': 15L}, {'author__count': 287, 'author': 14L}, {'author__count': 10, 'author': 6L}, {'author__count': 2101, 'author': 16L}]