Get emailed when a website changes

Klaxon is a free, quick to setup and easy to use service that checks websites for changes so you don't have to. Working with the fine folks at MuckRock and Document Cloud, we have launched an even easier version of Klaxon that requires no server for you to maintain and very little setup. We encourage all new individual users, or anyone who no longer wants to run their own pesky Heroku servers, to use the new Klaxon Cloud tool that is built into Document Cloud, the open document archive for journalists and researchers.

While the original version of Klaxon will remain free and open source on Github, we will no longer be supporting development for individual users. Feel free to fork the source code to run your own instance, and send back pull requests) for features you think will benefit the long-standing, institutional users.

In either variety of Klaxon, you add websites you want monitored and the tool will visit them periodically. If they change, it'll email you what's different. It's perfect for monitoring website changes you might miss — like freedom of information disclosure logs, court records or anything related to Donald Trump. It can even send notifications to Slack channels with a little extra setup.

Built and refined in the newsroom of The Marshall Project, Klaxon has provided our journalists with many news tips, giving us early warnings and valuable time to pursue stories. Klaxon has been used and tested by journalists at The Marshall Project, The New York Times, The Texas Tribune, The Washington Post, Associated Press, and more.

The documentation here refers solely to the original version of Klaxon that requires you to manage your own server and email service through Heroku. Read more about it below or say hello to the humans behind the project at the Google Group email list.

Alerting journalists to changes on the web

The public release of this free and open-source software was supported by Knight-Mozilla OpenNews.

How Does Klaxon Work?

Klaxon enables users to "bookmark" portions of a webpage and be notified (via email, Slack, or Discord) of any changes that may occur to those sections. Learn more about bookmarklets on the help.md page.

Setting up your Klaxon

Klaxon is open-source software built by the newsroom of The Marshall Project, a nonprofit investigative news organization covering the American criminal justice system. It was created by a team of three — Ivar Vong, Andy Rossback and Tom Meagher — and supported by contributions from users around the world. It is subject to the kind of shortcomings any side project might encounter. It may break unexpectedly. It may miss a change in a website or an email might not fire off correctly. Still we’ve found it immensely useful in our daily reporting. We want other journalists to benefit from Klaxon and to help us improve it, but keep these caveats in mind and use it at your own risk.

And if you’d like to help us make this better or add new functionality to it we’d love to have your help.

Getting started

One of our goals for Klaxon was to make it as easy as possible for reporters and editors without tech backgrounds to use and to setup. Getting your own Klaxon running in your newsroom will require you to run a handful of instructions one time through the help of online services Heroku and GitHub. It should take maybe 10 minutes to set up your Klaxon, including the time to create accounts on Heroku and GitHub if you need to.

We use Heroku to deploy software at The Marshall Project. We think it makes some of the tedious work of running servers a lot easier to deal with so we designed Klaxon to be easily deployable on Heroku. (If you’d like to run this in your newsroom’s preferred server setup — say using Docker or a Linux machine — we encourage you to do so, but know you'll be on your own maintaining it!)

If you want to use our setup, you’ll need to create an account with Heroku if you don’t already have one.

How much will this cost?

Heroku unfortunately no longer offers a free tier, so you will need to be able to pay around $10 a month to get started. Out of the box with Heroku you’ll get:

12,000 emails per month with SendGrid
10,000 records of changes in your Postgres database
A web interface
A scan of each watched site every 10 minutes

If you find your newsroom hitting the limits of these tiers you can pay to expand them. To send up to 40,000 emails a month you may upgrade your Sendgrid add-on in Heroku for $9.95 a month. If you need to store more records in your database you can pay an additional $4 a month for up to 10 million rows. And if you need your web interface running around the clock, you can upgrade your Heroku dyno from the Eco to the Hobby level for $2 more a month. This likely won't be necessary — particularly in smaller newsrooms — but it’s good to know.

Let’s do this

If you have a Heroku account you’re ready to go — it’s time to click on this button:

It will take you to a page to configure your new app in Heroku’s dashboard. First, give your app a name in the first box. While this is technically optional, this will also double as the URL for your Klaxon instance, so think carefully about it for a moment. Try maybe an abbreviation for your newsroom with a hyphen and the word klaxon, like “wp-klaxon” or “sl-klaxon”. This will become a URL as https://sl-klaxon.herokuapp.com/

Scroll down to the admin_emails field and add a comma-separated list of email addresses of your newsroom’s Klaxon administrators. These administrators will be able to create accounts for users in your organization as well as configure the Slack and Discord integrations.

Click the big purple “Deploy" button. If you haven’t given Heroku your credit card yet it will ask you for your information now. After that, give Heroku a few minutes for the app to build.

When you see this message:

...you’re almost done.

Click on the button that says “Manage App”. This takes you behind the scenes of the various components powering your Klaxon. On this resources screen, click on the link for “Heroku Scheduler,” which will take you a new screen where you must add the very important piece. The scheduler is what runs every 10 minutes to actually check all the sites and pages you’re watching. Click the long, purple "Add new job" button. In the text box next to the dollar sign, type the words “rake check:all” with the colon and without the quotes. Under “Frequency” change it from “Daily” to “Every 10 minutes”. Click the purple “Save” button and your scheduler item should look like this:

Sendgrid now requires additional steps to confirm that you are not a spammer. Your new Sendgrid account is created in a "suspended" state. To get it unsuspended you have to contact Sendgrid support. You can do this by clicking the Sendgrid logo on the Resources tab.

If clicking on the logo takes you to an error page do not worry. This has been known to happen as Sendgrid's system has undergone redesigns. Instead go to Sendgrid's page to ask for support. Be sure to use the same email address associated with your Heroku account and provide the url of your Klaxon instance. When they ask for "Business impact," choose "P3 General - You have a question about Sendgrid or how to use its products".

This step is a nuisance, but important. You will not be able to get an email to log in to Klaxon until you are cleared by Sendgrid. This usually happens pretty quickly.

Unfortunately you are not yet done configuring Sendgrid. There are more steps to set up your account.

From the Heroku application page, click on the "Resources" tab, and click the link to the Sendgrid plugin, this will take you to the Sendgrid website.
In the left hand column choose Settings -> API Keys. Click the blue "Create an API Key" button in the top right corner.
In the form that appears fill in the API key name (doesn't matter what you name it), make sure the "Full Access" option is selected, and click "Create & View".
Click the API key to copy it to your clipboard. Then navigate back to Heroku.
In the "Settings" tab of your Heroku app click the "Reveal Config Vars" button.
Change the SENDGRID_PASSWORD variable to the API Key by clicking the pencil icon next to it, pasting it in, and saving it.
Change the SENDGRID_USERNAME variable to the string "apikey" in the same manner.

Next you'll need to set up a "Verified Sender" account in Sendgrid using an email address that you have access to. In your Sendgrid dashboard, click "Sender Authentication" and choose "Verify Single Sender." (See https://github.com/themarshallproject/klaxon/issues/404 for some more context.)

When you've completed this process in Sendgrid, you'll need to set the MAILER_FROM_ADDRESS variable as you did a

Klaxon

Install / Use

README