This is the third post in the BibleFeed Project. If you haven’t already, read the first and second posts.
In my last post I stated the difficulty I was having finding a python library to handle the SOAP web service which I’ll be using to get the data for this project. I gave up on using a library for SOAP and decided to use urllib2 to send the SOAP request and retrieve the response, and ElementTree to parse the response. Both of these are standard libraries in Python 2.5 and higher, so you should not need to install anything extra to use these libraries.
Creating the SOAP request
I decided to take advantage of Django’s template system to create the SOAP requests. The advantages of this approach are that I can easily insert variable data into each SOAP request, I don’t have to manually build the XML in code using a potentially clumsy API, I’m not hard-coding the XML in a string, and tweaking the request is as simple as editing any other XML file.
To accomplish this, I created a templates directory under the bible directory (this the directory where models.py lives). I edited settings.py so that Django knows where the template directory is.
TEMPLATE_DIRS = (
# Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
'bible/templates',
)
In the templates directory I created a file called soaprequest_listbooks.xml which contains the SOAP request to get the list of books from the web service.
<SOAP-ENV:Envelope xmlns:ns0="http://www.francisshanahan.com/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header/>
<SOAP-ENV:Body>
<ns0:ListBooks/>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
You may have noticed that this template doesn’t use any variables at all. That’s okay, in the future I will be making this code usuable for all our SOAP requests, so I will need to be able to use template variables in the future.
Sending the SOAP request and reading the response
At this point, I haven’t written any code that does anything yet. I’m going to change that now. In the bible directory there is a mostly empty file called views.py. This file is intended to contain views, which are methods that produce an HttpResponse based on a given HttpRequest. I created a view called listbooks_view, which will retrieve the list of books of the Bible from the web service, and save these books in my database.
import urllib2
from django.template import Context, loader
from django.http import HttpResponse
from bible.models import *
from string import atoi
import xml.etree.ElementTree as ET
def listbooks_view(request):
# Create the SOAP request and send it
url = 'http://francisshanahan.com/TheHolyBible.asmx'
headers = {'Soapaction' : '"http://www.francisshanahan.com/ListBooks"',
'Content-Type' : 'text/xml'}
request_template = loader.get_template('soaprequest_listbooks.xml')
request_context = Context({}) #nothing is needed for this request
request_data = request_template.render(request_context)
http_req = urllib2.Request(url, request_data, headers)
http_resp = urllib2.urlopen(http_req)
# Assuming we got a successful response, parse it and store the results in the database
soap_resp = ET.fromstring(http_resp.read())
# Lovely path, huh?
books_xml = soap_resp.findall('{http://schemas.xmlsoap.org/soap/envelope/}Body/{http://www.francisshanahan.com/}ListBooksResponse/{http://www.francisshanahan.com/}ListBooksResult/{urn:schemas-microsoft-com:xml-diffgram-v1}diffgram/NewDataSet/bible_content')
for book_xml in books_xml:
id = atoi(book_xml.find('Book').text)
if id<100: # This webservice returns other stuff numbered 100 and higher that isn't actual bible content
book = Book()
book.id = id
book.name = book_xml.find('BookTitle').text
if id<40: #The first 39 books are in the Old Testament, the rest are New Testament
book.testament = 'O'
else:
book.testament = 'N'
book.save()
return HttpResponse('Success!')
The path used to parse the XML in the SOAP response is kind of nasty due to the heavy use of XML namespaces in the SOAP response. In my experience this is pretty common, and is just the nature of dealing with SOAP. To give you an idea of what the XML that I’m parsing looks like, here’s a snippet of the SOAP response:
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<ListBooksResponse xmlns="http://www.francisshanahan.com/">
<ListBooksResult>
<xs:schema id="NewDataSet" xmlns="" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata">
<xs:element name="NewDataSet" msdata:IsDataSet="true" msdata:UseCurrentLocale="true">
<xs:complexType>
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:element name="bible_content">
<xs:complexType>
<xs:sequence>
<xs:element name="Book" type="xs:int" minOccurs="0" />
<xs:element name="BookTitle" type="xs:string" minOccurs="0" />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:choice>
</xs:complexType>
</xs:element>
</xs:schema>
<diffgr:diffgram xmlns:msdata="urn:schemas-microsoft-com:xml-msdata" xmlns:diffgr="urn:schemas-microsoft-com:xml-diffgram-v1">
<NewDataSet xmlns="">
<bible_content diffgr:id="bible_content1" msdata:rowOrder="0">
<Book>1</Book>
<BookTitle>The First Book of Moses, called Genesis</BookTitle>
</bible_content>
<bible_content diffgr:id="bible_content2" msdata:rowOrder="1">
<Book>2</Book>
<BookTitle>The Second Book of Moses, Called Exodus</BookTitle>
</bible_content>
<bible_content diffgr:id="bible_content3" msdata:rowOrder="2">
<Book>3</Book>
If you’re following along at home, you may be tempting to run the django test web server and see if the code works. You will be disappointed when you see “ProgrammingError at /listbooks/ ERROR: value too long for type character varying(50).” What this means is that the name field in the Book model is too short. As you can see from the SOAP response above, this webservice uses long names for each of the books of the Bible. Where I initially expected names like “Matthew” and “Corinthians I”, instead I got names like “The Gospel According to St. Matthew” and “The First Epistle of Paul the Apostle to the Corinthians.” Thankfully this is easy to fix. I edited models.py so that the name field in the Book model is 120 characters long instead of 50.
class Book(models.Model):
TESTAMENTS = (
('O','Old Testament'),
('N','New Testament'),
)
name = models.CharField(max_length=120)
testament = models.CharField(max_length=1, choices=TESTAMENTS)
Next, I need to adjust the database so that the name column for the “bible_book” table is 120 characters long. Note that in the snippet below I use PostgreSQL for my database. If you are using MySQL or some other database, the SQL will be slightly different.
$ python manage.py dbshell
Password for user postgres:
Welcome to psql 8.3.3, the PostgreSQL interactive terminal.
Type: \copyright for distribution terms
\h for help with SQL commands
\? for help with psql commands
\g or terminate with semicolon to execute query
\q to quit
biblefeed=# alter table bible_book alter name type varchar(120);
ALTER TABLE
biblefeed=# \d bible_book
Table "public.bible_book"
Column | Type | Modifiers
-----------+------------------------+---------------------------------------------------------
id | integer | not null default nextval('bible_book_id_seq'::regclass)
name | character varying(120) | not null
testament | character varying(1) | not null
Indexes:
"bible_book_pkey" PRIMARY KEY, btree (id)
Now, you can run the django test webserver. Open http://localhost:8000/listbooks/ and you should see “Success!”
So how do I know this worked? I go back to the database to see what’s in the “bible_book” table.
$ python manage.py dbshell
Password for user postgres:
Welcome to psql 8.3.3, the PostgreSQL interactive terminal.
Type: \copyright for distribution terms
\h for help with SQL commands
\? for help with psql commands
\g or terminate with semicolon to execute query
\q to quit
biblefeed=# select * from bible_book order by id;
id | name | testament
----+-------------------------------------------------------------+-----------
1 | The First Book of Moses, called Genesis | O
2 | The Second Book of Moses, Called Exodus | O
3 | The Second Book of Moses, called Leviticus | O
4 | The Fourth Book of Moses, called Numbers | O
5 | The Fifth Book of Moses, called Deuteronomy | O
6 | The Book of Joshua | O
7 | The Book of Judges | O
8 | The Book of Ruth | O
9 | The First Book of Samuel | O
10 | The Second Book of Samuel | O
11 | The First Book of the Kings | O
12 | The Second Book of the Kings | O
13 | The First Book of the Chronicles | O
14 | The Second Book of the Chronicles | O
15 | The Book of Ezra | O
16 | The Book of Nehemiah | O
17 | The Book of Esther | O
18 | The Book of Job | O
19 | The Book of Psalms | O
20 | The Proverbs | O
21 | Ecclesiastes or, The Preacher | O
22 | The Song of Songs, Which is Solomon's | O
23 | The Book of the Prophet Isaiah | O
24 | The Book of the Prophet Jeremiah | O
25 | The Lamentations of Jeremiah | O
26 | The Book of the Prophet Ezekiel | O
27 | The Book of Daniel | O
28 | The Book of Hosea | O
29 | The Book of Joel | O
30 | The Book of Amos | O
31 | The Book of Obadiah | O
32 | The Book of Jonah | O
33 | The Book of Micah | O
34 | The Book of Nahum | O
35 | The Book of Habakkuk | O
36 | The Book of Zephaniah | O
37 | The Book of Haggai | O
38 | The Book of Zechariah | O
39 | The Book of Malachi | O
40 | The Gospel According to St. Matthew | N
41 | The Gospel According to Saint Mark | N
42 | The Gospel According to St. Luke | N
43 | The Gospel According to Saint John | N
44 | The Acts of the Apostles | N
45 | The Epistle of Paul the Apostle to the Romans | N
46 | The First Epistle of Paul the Apostle to the Corinthians | N
47 | The Second Epistle of Paul the Apostle to the Corinthians | N
48 | The Epistle of Paul the Apostle to the Galatians | N
49 | The Epistle of Paul the Apostle to the Ephesians | N
50 | The Epistle of Paul the Apostle to the Philippians | N
51 | The Epistle of Paul the Apostle to the Colossians | N
52 | The First Epistle of Paul to the Thessalonians | N
53 | The Second Epistle of Paul the Apostle to the Thessalonians | N
54 | The First Epistle of Paul the Apostle to Timothy | N
55 | The Second Epistle of Paul the Apostle to Timothy | N
56 | The Epistle of Paul to Titus | N
57 | The Epistle of Paul to Philemon | N
58 | The Epistle to the Hebrews | N
59 | The General Epistle of James | N
60 | The First Epistle General of Peter | N
61 | The Second Epistle General of Peter | N
62 | The First Epistle General of John | N
63 | The Second Epistle of John | N
64 | The Third Epistle of John | N
65 | The General Epistle of Jude | N
66 | The Revelation to Saint John | N
(66 rows)
If you have questions, or if I missed something or just didn’t cover it enough, then leave a comment.