Managing Your Academic Library with Zotero and Apache mod_dav

Introduction: Why Zotero?

In the academic world, efficient management of literature is often key to avoiding chaos and enabling productive work. Today its often supplemented by artificial intelligence - for example LLMs as literature research tools. Zotero is an open-source reference management software that excels at collecting, organizing, citing, and sharing research sources. Unlike proprietary tools, Zotero is free, supports tagging and note-taking, stores local PDFs alongside structured bibliographic data, can generate BibTeX databases for your LaTeX documents (which remains the only sane word processor for academic work out there), keeps track of relations between documents if you enter them (e.g. citations, responses, related works), and allows imports directly from BibTeX or by supplying a handle, DOI, or PMID, making it an ingenious all-in-one academic reference and PDF management tool.

Zotero’s search feature is incredibly powerful. It can search not only titles and authors but full-text content of indexed PDFs, tags, notes, and related metadata, allowing you to instantly locate any article or note you need, even within massive libraries, using keyword, phrase, or advanced field queries.

Zotero offers a robust sync feature that allows synchronizing your entire Zotero database - including collections, tags, metadata, and notes - via their official sync service (the providers database). This enables seamless access across different machines. Unfortunately, at the moment, there is no (easy) way to substitute this central database sync with one’s own server, but it remains crucial to keep library data consistent between devices. Though the source code for the server is open the functionality of switching metadata servers is not simply exposed in the configuration.

However, many users struggle with limited cloud storage for PDF (or other) attachments when using Zotero’s official sync. The solution: either pay for their (not so expensive - but we know many small services sum up when you don’t get it funded by your institution) additional storage space plans or hosting your own WebDAV storage to keep all your PDFs accessible and synced across devices without paying for premium storage - and without having to upload potentially copyrighted material to a third party which may be (depending on the institution you are working on) be a problem. However, Zotero stores its data in its own proprietary format inside prop and zip files, so one cannot easily attach external tools or use a direct external directory view - but despite this limitation, it remains a very valuable tool for literature management.

Due to its open source character Zotero is available:

• On FreeBSD as the zotero package that can be installed directly via pkg install zotero
• On many Linux distributions as package
• Windows
• Android
• iOS
• MacOS

In addition one can use the excellent browser connectors to have a one-click solution to store citation information and full text that one can access directly from the browser into ones literature database as long as the application is running in the background.

• What is WebDAV and Why Use It with Zotero?
• Setting Up Your Own WebDAV Server with Apache httpd and mod_dav
  • Install Required Modules
  • Create a Directory for Zotero Files
  • Configure Apache VirtualHost or Location
  • Create HTTP Basic Auth Credentials
  • Reload or restart Apache
• Configure Zotero to Use Your WebDAV Server
• Conclusion: Power and Independence

What is WebDAV and Why Use It with Zotero?

WebDAV (Web Distributed Authoring and Versioning) is a protocol that allows remote management of files over HTTP. It builds upon basic HTTP, which already included methods like GET, POST, PUT, DELETE, and HEAD to allow uploading, modifying, and deleting resources directly, by adding new methods and structured metadata support. WebDAV was developed in the late 1990s as an extension of HTTP/1.1, adding methods such as PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK, and UNLOCK to support collaborative authoring, locking and file manipulation. WebDAV exposes a hierarchical file system model over HTTP. Access control is usually performed via HTTP basic or digest authentication combined with server-based permission management.

A related extension is Delta-V (WebDAV Versioning), which defined versioning, check-in/check-out, and workspace schemes; unfortunately there is no full Delta-V implementation as of today, with Subversion having been the closest and most standard conforming at the time of writing, as most servers implement only small parts of the specification.

Zotero supports WebDAV - without versioning - as a backend for syncing PDF attachments while metadata is synced via Zotero’s own servers. This means you can:

• Keep PDFs and other attachments stored on your own server
• Save money by avoiding external cloud plans
• Access the same library across multiple devices securely

Setting Up Your Own WebDAV Server with Apache httpd and mod_dav

Here is a short tutorial to get your personal WebDAV storage running under Apache. This assumes you have a FreeBSD server with Apache 2.4 installed and familiarity with server administration. On Linux and Windows this works similar, you most likely just have to adjust the paths.

Install Required Modules

We are using mod_dav as WebDAV interface with the mod_dav_fs backend that allows to access the servers filesystem to be accessed via WebDAV. In addition we require the mod_dav_lock module since Zotero uses file locking to synchronize operations over different clients.

First ensure mod_dav, mod_dav_fs and mod_dav_lock are enabled in your /usr/local/etc/apache24/httpd.conf:

LoadModule dav_module libexec/apache24/mod_dav.so
LoadModule dav_fs_module libexec/apache24/mod_dav_fs.so
LoadModule dav_lock_module libexec/apache24/mod_dav_lock.so

Check if everything is ok by gracefully restarting the server afterwards

$ apachectl graceful

Create a Directory for Zotero Files

In this article I’m assuming you have a specific directory structure where you store all configurations, data and webroots under a hierarchie located at /usr/www/{domainname}/:

• The webroot at /usr/www/{domainname}/www/
• The configuration at /usr/www/{domainname}/conf/
• The associated data at /usr/www/{domainname}/data/

I also assume we have an www__domainname user and group for the given virtual host to provide isolation.

If you do this differently keep in mind you have to adjust the paths accordingly.

First we have to create a zotero folder. The name is not changeable but you can put it anywhere in your hierarchy. I assume we use our own virtual host - but you can add this to any VirtualHost or the default host at any point using very similar commands. To create the folder:

mkdir /usr/www/www.example.com/www/zotero
chown www__www_example_com /usr/www/www.example.com/www/zotero
chgrp www__www_example_com /usr/www/www.example.com/www/zotero
chmod 770 /usr/www/www.example.com/www/zotero

Make sure the user and group that the webserver is running with is actually capable of accessing this folder with read and write permissions. The same goes for the /usr/www/www.example.com/data folder that we are going to use for the lock database.

Configure Apache VirtualHost or Location

Add the following to your VirtualHost configuration or your global configuration to configure the lock database. This cannot be added to a Directory or Location:

DAVLockDB /usr/www/www.example.com/data/davlock

The name davlock is arbitrarily choosen. You can use any. Then create the Directory entry for the zotero directory:

<Directory "/usr/www/www.example.com/www/zotero">
   DAV On

   # Actually its better to use Digest authentication of SSL
   # client certificates though ... we show this example with
   # basic auth. Make at least sure you use SSL when doing this.

   AuthType Basic
   AuthName "Example DAV Realm"
   AuthUserFile /usr/www/www.example.com/conf/users.htpasswd

   Require valid-user
</Directory>

This enabled WebDAV on the zotero folder and requires users to be authenticated before they are allowed to perform operations on the directories.

Create HTTP Basic Auth Credentials

When using HTTP basic authentication we have to generate a htpasswd file using the htpasswd tool. Keep in mind this is the weakest authentication method. Digest authentication works better and can actually work together with single sign on solutions like Kerberos - Client SSL certificates would provide the best security though. Whenever you use Basic auth you at least make sure to set up SSL correctly - when using Basic the password goes to the server in plain text (this is mitigated with digest authentication).

On the first time create the password file:

htpasswd -c /usr/www/www.example.com/conf/users.htpasswd yourusername

After the initial one you can add users or change passwords leaving out the -c argument using

htpasswd /usr/www/www.example.com/conf/users.htpasswd yourusername

Reload or restart Apache

apachectl graceful

Configure Zotero to Use Your WebDAV Server

In Zotero preferences, go to:

• Switch to the Sync tab
• Under File Syncing, choose WebDAV for sync attachment files in My Library. This displays an dialog that allows you to enter the URL, username and password.
• Enter the URL, e.g. https://www.example.com/zotero. Please really use SSL.
• Enter your username and password
• Click Verify Server.

If everything turns out correct you can also enable to sync attachment files into group libraries in case you like to use this feature of the official file repository.

That’s it, syncing works from now on. Do this configuration on every machine you are working on.

Conclusion: Power and Independence

Using Zotero with your own WebDAV server gives you full control over your research library’s storage, ensures privacy, helps you to stay within legal constraints and integrates seamlessly with your workflows. Though there are always improvements possible (I could imagine having an easily externally accessable structure in the WebDAV share, having the ability to run ones own database instead of the provided official one, etc.) Zotero provides a highly valueable Open-Source tool that is available for many different platforms that helps one to keep research organized.

This article is tagged:

• Tutorial
• Web
• Administration
• System administration
• FreeBSD
• Basics