Stylometry and the Pauline Epistles

Stylometry is the statistical analysis of literary style. It includes the study of documents’ quantitative attributes like average sentence length or relative frequencies of certain words. Authorship attribution is an important application of stylometry. Individual authors have stylometric fingerprints, and those fingerprints can help to determine who wrote (or didn’t write) certain documents.

A famous authorship attribution problem concerns The Federalist Papers, a collection of essays by Alexander Hamilton, John Jay, and James Madison. All three men published essays under the common pseudonym “Publius.” Because of this, it was unknown for some time which of them had written some of the essays. In 1963, statisticians Frederick Mosteller and David Wallace analyzed word frequencies to determine the authorship of the disputed papers . They made the striking observation that Hamilton used the word “upon” about 18 times more frequently than did Madison. This feature, among many others, allowed Mosteller and Wallace to conclude that Madison had written twelve of the disputed essays.

The New Testament is another interesting candidate for stylometric analysis. Its twenty-seven books were written by several authors; some known, some completely anonymous and unknown, and some contested. Of special interest are the Pauline epistles: thirteen letters to churches or individuals that bear the Apostle Paul’s name. Scholarly consensus regards seven of these epistles (Romans, 1 and 2 Corinthians, Galatians, Philippians, 1 Thessalonians, and Philemon) as genuinely Pauline. The authenticity of the others (Ephesians, Colossians, 2 Thessalonians, 1 and 2 Timothy, and Titus) is disputed. Some church traditions consider Hebrews to be Pauline, but it is anonymous and scholars agree that Paul didn’t write it.

Burrows’s Delta

Burrows’s Delta is a statistic that measures similarity between two texts . Roughly speaking, it compares the relative frequencies of certain words in each text. It is defined as follows.

Let and be two documents within a larger database of documents and let be the most frequently occurring words in the database. We will denote by the number of times that appears in . In order to treat commonly appearing words with the same weight as less common words, Burrows used -scores to normalize word counts.

Here and are the average and standard deviation, respectively, of over all documents in the database. Delta is then defined as follows.

Burrows originally used his Delta to match a disputed text to one author from a fixed set of possible authors. Our situation is a bit different: we’re trying to determine whether or not a single person is likely to be the author of certain texts; there is no fixed list of candidate authors. Even though our implementation will be different than Burrows’s, we should still be able to extract some meaningful information.

Our database of documents will be the collection of New Testament books (or, in some cases, groups of New Testament books by the same author). The most important document in our database will be a single Pauline “training text” to which other New Testaments books can be compared. I randomly chose four of Paul’s undisputed epistles (Romans, 1 Corinthians, Galatians, and Philemon) to serve as the Pauline training text. Those books, together, will be our baseline for evaluating Paul’s writing style. His remaining three undisputed epistles (2 Corinthians, Philippians, and 1 Thessalonians) will be left as a sort of control group. By measuring Delta between the control group epistles and our Pauline training text, we’ll get an idea of what Delta values to expect from authentic Pauline literature. We’ll also measure Delta between non-Pauline New Testament works and the Pauline training text to get an idea of what Delta values to expect from other authors.

Preparing the Text for Analysis

A complete Greek New Testament is available in from http://www.sacred-texts.com/bib/osrc/. Let’s see how it looks.

lines = []
with open('ugntdat.txt', encoding='utf8') as f:
    for line in f:
        line = line.strip()
        lines.append(line)

for line in lines[:10]:
    print(line)

Mat|1|1|βίβλος γενέσεως ἰησοῦ χριστοῦ υἱοῦ δαυὶδ υἱοῦ ἀβραάμ.
Mat|1|2|ἀβραὰμ ἐγέννησεν τὸν ἰσαάκ, ἰσαὰκ δὲ ἐγέννησεν τὸν ἰακώβ, ἰακὼβ δὲ ἐγέννησεν τὸν ἰούδαν καὶ τοὺς ἀδελφοὺς αὐτοῦ,
Mat|1|3|ἰούδας δὲ ἐγέννησεν τὸν φάρες καὶ τὸν ζάρα ἐκ τῆς θαμάρ, φάρες δὲ ἐγέννησεν τὸν ἑσρώμ, ἑσρὼμ δὲ ἐγέννησεν τὸν ἀράμ,
Mat|1|4|ἀρὰμ δὲ ἐγέννησεν τὸν ἀμιναδάβ, ἀμιναδὰβ δὲ ἐγέννησεν τὸν ναασσών, ναασσὼν δὲ ἐγέννησεν τὸν σαλμών,
Mat|1|5|σαλμὼν δὲ ἐγέννησεν τὸν βόες ἐκ τῆς ῥαχάβ, βόες δὲ ἐγέννησεν τὸν ἰωβὴδ ἐκ τῆς ῥούθ, ἰωβὴδ δὲ ἐγέννησεν τὸν ἰεσσαί,
Mat|1|6|ἰεσσαὶ δὲ ἐγέννησεν τὸν δαυὶδ τὸν βασιλέα. δαυὶδ δὲ ἐγέννησεν τὸν σολομῶνα ἐκ τῆς τοῦ οὐρίου,
Mat|1|7|σολομὼν δὲ ἐγέννησεν τὸν ῥοβοάμ, ῥοβοὰμ δὲ ἐγέννησεν τὸν ἀβιά, ἀβιὰ δὲ ἐγέννησεν τὸν ἀσάφ,
Mat|1|8|ἀσὰφ δὲ ἐγέννησεν τὸν ἰωσαφάτ, ἰωσαφὰτ δὲ ἐγέννησεν τὸν ἰωράμ, ἰωρὰμ δὲ ἐγέννησεν τὸν ὀζίαν,
Mat|1|9|ὀζίας δὲ ἐγέννησεν τὸν ἰωαθάμ, ἰωαθὰμ δὲ ἐγέννησεν τὸν ἀχάζ, ἀχὰζ δὲ ἐγέννησεν τὸν ἑζεκίαν,
Mat|1|10|ἑζεκίας δὲ ἐγέννησεν τὸν μανασσῆ, μανασσῆς δὲ ἐγέννησεν τὸν ἀμώς, ἀμὼς δὲ ἐγέννησεν τὸν ἰωσίαν,

Each verse is conveniently formatted as (abbreviated book title)|(chapter number)|(verse number)|(verse text). We only need the book titles and the individual words in each book, so we’ll make a dictionary that associates titles with texts. Each dictionary value will be a list of words. Python’s string module will help us get rid of all the punctuation.

import string

# Make a list of New Testament book abbreviations.
books = []
for line in lines:
    if line[:3] not in books:
        books += [line[:3]]

# Make a dictionary of word lists from New Testament books.
book_dict = {}
for line in lines:
    if line[:3] not in book_dict:
        book_dict[line[:3]] = []
    book_dict[line[:3]] += line.split('|')[-1].translate(str.maketrans('', '', string.punctuation+'·')).split()
    
print(book_dict['Mat'][:10])

['βίβλος', 'γενέσεως', 'ἰησοῦ', 'χριστοῦ', 'υἱοῦ', 'δαυὶδ', 'υἱοῦ', 'ἀβραάμ', 'ἀβραὰμ', 'ἐγέννησεν']

Computations

Now that the data is in a usable format, we can start counting words. We’ll make a list of features, that is, distinct words. Burrows’s Delta uses only the most frequent words, so we’ll only include words that appear in the New Testament at least twice. That’s probably more words than we need, but we can worry about choosing later.

# Make a dictionary of word counts of distinct New Testament words.
word_count_dict = {}
for book, text in book_dict.items():
    for word in text:
        if word in word_count_dict:
            word_count_dict[word] += 1
        else:
            word_count_dict[word] = 1

import nltk

NT_word_list = []

for book in books:
    NT_word_list += book_dict[book]

# Make a list of every word in the New Testament that appears at least twice.
features = [x[0] for x in nltk.FreqDist(NT_word_list).most_common() if x[1] >= 2]

for feature in features[:10]:
    print(feature)

καὶ
ὁ
ἐν
δὲ
τοῦ
εἰς
τὸ
τὸν
τὴν
αὐτοῦ

Now we’ll group some New Testament works together. Romans, 1 Corinthians, Galatians, and Philemon were chosen to be our Pauline training text, so we’ll group those books together. I also chose to group John’s three epistles together. 2 John and 3 John are very short (less than 250 words each), and shorter documents are more likely to have uncharacteristic word frequencies. Combining them with 1 John avoids this problem.

# Make a dictionary that groups together some books by author.
group_dict = {'Rom-Co1-Gal-Plm':[], 'Jo1-2-3':[]}
for book in books:
    if book in ['Rom', 'Co1', 'Gal', 'Plm']:
        group_dict['Rom-Co1-Gal-Plm'] += book_dict[book]
    elif book in ['Jo1', 'Jo2', 'Jo3']:
        group_dict['Jo1-2-3'] += book_dict[book]
    else:
        group_dict[book] = book_dict[book]

Now we can make a Pandas DataFrame with word frequencies for each book or group of books.

import pandas as pd

# Make a dataframe with word occurrence counts for each group of New Testament books.
df = pd.DataFrame(columns=features)

for group, text in group_dict.items():
    group_len = len(group_dict[group])
    for feature in features:
        df.at[group, feature] = group_dict[group].count(feature)/group_len

df.head()

Note that the word “καὶ” accounts for about 3.9% of all the words in our Pauline training text. Other books use the word much more frequently. Burrows’s Delta will be able to detect this and other differences between texts. Next, we’ll normalize the data and write a function to compute Burrows’s Delta.

z_df = (df-df.mean())/df.std()

z_df.head()

def Delta(A, B, n):
    # A and B must be z_df indices, e.g., A='Mat'.
    return sum(abs(z_df.loc[A].iloc[:n]-z_df.loc[B].iloc[:n]))/n

Choosing the Number of Words

Burrows’s Delta depends on the number of most frequent words. Choosing is an inexact science. The best we can do is pick one that seems to work. To this end, we’ll try out many different values of and see if any are able to distinguish between Pauline and non-Pauline texts.

Authentic_Paul = ['Co2', 'Phi', 'Th1']
Not_Paul = ['Mat', 'Mar', 'Luk', 'Joh', 'Act', 'Heb', 'Jam', 'Pe1', 'Pe2', 'Jo1-2-3', 'Jde', 'Rev']
Disputed_Paul = ['Eph', 'Col', 'Th2', 'Ti1', 'Ti2', 'Tit']

Deltas_df = pd.DataFrame(columns=Authentic_Paul+Not_Paul+Disputed_Paul)

for D in Authentic_Paul+Not_Paul+Disputed_Paul:
    for n in range(100, len(features), 100):
        Deltas_df.at[n, D] = Delta('Rom-Co1-Gal-Plm', D, n)

Deltas_df.head()

import plotly.plotly as py
import cufflinks as cf

dashes = ['solid' for D in Authentic_Paul] + ['dash' for D in Not_Paul]

Deltas_df[Authentic_Paul+Not_Paul].iplot(kind='scatter',
    title='Burrows\'s Delta between New Testament Books and a Pauline Training Text',
    xTitle='Number of Most Frequent Words', yTitle='Burrows\'s Delta',
    filename='pauline_epistles_fig1', dash=dashes)

There’s a lot going on in that graph, so let’s take some time to break it down. First of all, notice that Delta values for the four gospels (Matthew, Mark, Luke, and John) along with Acts and Revelation all begin to diverge at around 1200 words. Those six books are the only New Testament books that are not epistles, so Delta may be detecting a difference in content or genre between epistles and non-epistles. Second, it appears that Delta is able to distinguish between Pauline and non-Pauline epistles when is between about 2000 and 2600. You can zoom in on the graph to see this up close. Note that Deltas for authentic Pauline epistles are plotted with solid lines and Deltas for non-Pauline works are plotted with dashed lines.

Results

It looks like 2300 is a good choice of , so let’s compare Delta values for the disputed Pauline epistles to Delta values for other books using the 2300 most frequent New Testament words.

Deltas_df[Authentic_Paul].loc[2300].sort_values()

Co2     0.58551
Th1    0.656688
Phi    0.656939
Name: 2300, dtype: object

Deltas_df[Not_Paul].loc[2300].sort_values()

Pe1        0.674724
Pe2        0.678179
Jde        0.686343
Heb        0.709118
Jam        0.715614
Jo1-2-3     0.73883
Act        0.895024
Luk        0.920648
Rev        0.931996
Joh        0.934643
Mat         1.02118
Mar         1.05384
Name: 2300, dtype: object

Deltas_df[Disputed_Paul].loc[2300].sort_values()

Eph    0.650448
Col    0.654166
Th2    0.660886
Tit    0.665933
Ti2    0.683839
Ti1    0.686919
Name: 2300, dtype: object

Ephesians and Colossians are both more similar to our Pauline training text than 1 Thessalonians and Philippians. This indicates that Ephesians and Colossians were both likely written by Paul. 2 Thessalonians and Titus are both more similar to our Pauline training text than any non-Pauline texts, so Pauline authorship also appears plausible for those two books. 1 and 2 Timothy, on the other hand, are both less similar to the Pauline training text than both 1 and 2 Peter (and in the case of 1 Timothy, even less similar than Jude). This is evidence against Pauline authorship, although it’s far from conclusive.

Further Discussion

There are some drawbacks to the methods employed here. First of all, I do not know Greek. Naturally, this limits my ability to analyse Greek text. One issue that I don’t know how to handle is the presence of different diacritics on words that appear to be otherwise identical. For example, the name “Abraham” appears in Matthew 1:1 (“ἀβραάμ”) and Matthew 1:2 (“ἀβραὰμ”), but with a different diacritic above the final α in each case. Since these are distinct strings in Python, my analysis considers them to be distinct words. I have no idea if these kinds of differences are helpful or unhelpful for authorship determination.

Other complications arise from the messiness of New Testament authorship. In an ideal world, I would have grouped New Testament books by author (as I did with 1, 2, and 3 John). This would have prevented a single author from exerting undo force on word frequency averages, and it would have avoided the problem of comparing any one author to Paul more than once. However, it is impossible to make perfect delineations between New Testament authors for at least two reasons. First, there is debate over authorship of 1 and 2 Peter and over authorship of John’s Gospel, John’s Epistles, and Revelation. Second, some New Testament books blur authorship lines by reusing material from other books. Matthew and Luke, for example, both copied content from Mark, and Jude contains material that is very close to parts of 2 Peter (hence the similar Deltas between those two books and our Pauline training text).

Despite these difficulties, my naive approach was sensitive enough to distinguish between known categories (the undisputed Pauline epistles and the non-Pauline epistles). Hopefully this demonstrates the power of stylometric analysis.