hugh/ausglamr
1
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: hugh:main
Branches Tags
hugh:main
hugh:fixbyline
hugh:admin
hugh:dependabot/pip/django-4.2.14
hugh:docs
hugh:fix-summaries
hugh:hughrun-patch-1
hugh:editions
hugh:fixes
..
compare: hugh:hughrun-patch-1
Branches Tags
hugh:main
hugh:fixbyline
hugh:admin
hugh:dependabot/pip/django-4.2.14
hugh:docs
hugh:fix-summaries
hugh:hughrun-patch-1
hugh:editions
hugh:fixes

No commits in common. "main" and "hughrun-patch-1" have entirely different histories.
main ... hughrun-pa

19 changed files with 74 additions and 208 deletions
Show stats Expand all files Collapse all files

3
.gitignore vendored
View file

@ -1,6 +1,5 @@
/data
fixtures
/static
.env
.env.dev
z_README_Hugh.md
z_README_Hugh.md

59
README.md
View file

@ -5,18 +5,17 @@ A django app running on Docker. Replaces _Aus GLAM Blogs_.
## Deploy
* `cp .env.example .env` and enter env values for your app
* set up web server config (see nginx example)
* set up web server config (nginx example coming soon)
* `docker compose build`
* `./glamr-dev migrate` (you may get an error at this point - ignore)
* `./glamr-dev migrate`
* `./glamr-dev createsuperuser`
* `./glamr-dev collectstatic`
* copy static files to where your webserver can find them, e.g.`/srv/ausglamr`
* `docker compose up -d`
* set up cron jobs for management commands as outlined below
* set up database backups (as cron jobs): `./glamr-dev backup`:
* set up cron jobs for management commands as below
## Admin
Don't forget to add some **Content Warnings** for use by the Mastodon bot, within `/admin`.
Don't forget to add some Content Warnings for use by the Mastodon bot, within `/admin`.
## CLI tool
@ -60,9 +59,9 @@ These will not be triggered within the app - they should be called via cron jobs
### announce
This posts the next queued announcement on Mastodon.
This announces the next queued announcement on Mastodon.
Run about every 20 mins.
Run every 21 mins.
### check_feeds
@ -84,46 +83,8 @@ Does what you think. Creates a weekly email of the latest stuff, and send to eve
Run weekly.
## Backups
### Backups
### Creating backups
There is a `backup` command in `glamr-dev`. You can adjust the filepaths in your `.env` file.
There is a `backup` command in `glamr-dev`. The backup is named after the current day of the week, so in effect there will be a maximum of 7 rotating backups. The backup command relies on two environment variables: `DOCKER_PATH` and `BACKUPS_DIR` - you need to set these in your `.env` if you plan to run backups manually, but if you are running it with a cron job (recommended) you need to set environment variables within the crontab itself. See more details below.
### Restoring backups
1. back up your VPS if possible (e.g. taking a snapshot)
2. Locate the latest backup, or run the backup program: `./glamr-dev backup`
3. copy database dump into the container: `docker cp ~/ausglamr.dump ausglamr-db-1:/tmp/`
4. restore the dump: `docker exec -d ausglamr-db-1 pg_restore -c -e -U ausglamr -d ausglamr /tmp/ausglamr.dump`
## Cron jobs
The obvious way to run the management and backup commands is via cron jobs. It's important to note that environment variables generally need to be set within the crontab - you can't rely on passing them in from Ausglamr nor from a user context. If you forget to to this they will be blank and one of the bad things that will happen is cron will try and fail to run the Unix program `exec` instead of `docker exec` when trying to run backups.
Below are suggested crontab settings and jobs for Ausglamr.
```
# Aus GLAMR
# ---------
# envs
PATH=/bin:/usr/bin:/usr/local/bin:/snap/bin
DOCKER_PATH=/usr/bin/docker
BACKUPS_DIR=/home/ausglamr/backups
# Announce thrice an hour
1,23,47 * * * * cd /home/ausglamr/ausglamr && /home/ausglamr/ausglamr/glamr-dev announce
# Check feeds hourly on the eleventh minute
11 * * * * cd /home/ausglamr/ausglamr && /home/ausglamr/ausglamr/glamr-dev check_feeds
# Queue event announcements daily at 4:01am
1 4 * * * cd /home/ausglamr/ausglamr && /home/ausglamr/ausglamr/glamr-dev queue_announcements
# Run backups daily - 1:01am
1 1 * * * cd /home/ausglamr/ausglamr && /home/ausglamr/ausglamr/glamr-dev backup
# Send email weekly at 12:05pm
5 12 * * 1 cd /home/ausglamr/ausglamr && /home/ausglamr/ausglamr/glamr-dev send_weekly_email
```
Run daily

7
ausglamr/urls.py
View file

@ -1,7 +1,6 @@
"""
URL configuration for ausglamr project.
"""
from django.contrib import admin
from django.urls import path, re_path
from django.views.generic import TemplateView
@ -66,8 +65,6 @@ urlpatterns = [
views.UnsubscribeEmail.as_view(),
name="unsubscribe-email",
),
path(
"robots.txt",
TemplateView.as_view(template_name="robots.txt", content_type="text/plain"),
),
path('robots.txt', TemplateView.as_view(template_name='robots.txt',
content_type='text/plain')),
]

8
blogs/admin.py
View file

@ -131,7 +131,6 @@ class Blog(admin.ModelAdmin):
"active",
)
ordering = ["approved", "-suspended", "-failing"]
search_fields = ["title", "author_name", "url"]
actions = [approve, unapprove, suspend, unsuspend, activate, disable]
@ -141,7 +140,6 @@ class Article(admin.ModelAdmin):
date_hierarchy = "pubdate"
list_display = ("title", "blog_title", "pubdate")
search_fields = ["title", "author_name", "blog_title", "url"]
def blog_title(self, obj): # pylint: disable=no-self-use
"""get the title of the parent blog"""
@ -153,7 +151,6 @@ class Tag(admin.ModelAdmin):
"""display settings for tags"""
list_display = ("name",)
search_fields = ["name"]
@admin.register(models.Event)
@ -169,7 +166,6 @@ class Event(admin.ModelAdmin):
"start_date",
)
ordering = ["approved", "announcements"]
search_fields = ["name", "description", "url"]
actions = [approve, unapprove]
@ -180,7 +176,6 @@ class CallForPapers(admin.ModelAdmin):
list_display = ("name", "event", "approved", "closing_date")
list_select_related = ("event",)
ordering = ["approved", "closing_date"]
search_fields = ["event__name", "event__url", "details"]
actions = [approve, unapprove]
@ -190,7 +185,6 @@ class Group(admin.ModelAdmin):
list_display = ("name", "approved", "category", "description")
ordering = ["approved", "name"]
search_fields = ["name", "description", "url"]
actions = [approve, unapprove]
@ -206,7 +200,6 @@ class Newsletter(admin.ModelAdmin):
"active",
)
ordering = ["approved", "-failing"]
search_fields = ["name", "description", "url", "author_name"]
actions = [approve, unapprove, suspend, activate, disable]
@ -216,7 +209,6 @@ class Edition(admin.ModelAdmin):
date_hierarchy = "pubdate"
list_display = ("title", "newsletter_name", "pubdate")
search_fields = ["title", "description", "url", "newsletter__name", "author_name"]
def newsletter_name(self, obj): # pylint: disable=no-self-use
"""get the title of the parent newsletter"""

1
blogs/apps.py
View file

@ -1,5 +1,4 @@
"""django apps"""
from django.apps import AppConfig

4
blogs/forms.py
View file

@ -141,8 +141,8 @@ class ContactForm(forms.Form):
bot_check = forms.CharField(
label="What is usually stored in a library?",
max_length=10,
help_text="Checking that you are human",
)
help_text="Checking that you are human"
)
def clean_bot_check(self):
"""validate the bot check"""

68
blogs/management/commands/check_feeds.py
View file

@ -117,7 +117,7 @@ class Command(BaseCommand):
if not opt_out:
author_name = getattr(
article, "author", None
) or getattr(blog, "author", "")
) or getattr(blog, "author", None)
description = (
html.strip_tags(article.summary)
@ -125,32 +125,26 @@ class Command(BaseCommand):
hasattr(article, "summary")
and len(article.summary) > 0
)
else (
html.strip_tags(article.description)
if (
hasattr(article, "description")
and len(article.summary)
)
else (
html.strip_tags(article.content[0].value)
if (
hasattr(article, "content")
and len(article.content)
)
else None
)
else html.strip_tags(article.description)
if (
hasattr(article, "description")
and len(article.summary)
)
else html.strip_tags(article.content[0].value)[:200]
if (
hasattr(article, "content")
and len(article.content)
)
else ""
)
if description:
desc = description[:200] + "..."
else:
desc = ""
description += "..."
instance = models.Article.objects.create(
title=article.title,
author_name=author_name,
url=article.link,
description=desc,
description=description,
updateddate=date_to_tz_aware(
article.updated_parsed
),
@ -174,9 +168,7 @@ class Command(BaseCommand):
if newish:
instance.announce()
blog.set_success(
updateddate=date_to_tz_aware(
article.updated_parsed
)
updateddate=date_to_tz_aware(article.updated_parsed)
)
except Exception as e:
@ -199,7 +191,7 @@ class Command(BaseCommand):
| Q(guid=getattr(edition, "id", edition.link))
).exists():
author_name = getattr(edition, "author", None) or getattr(
edition, "author", ""
edition, "author", None
)
description = (
@ -207,32 +199,26 @@ class Command(BaseCommand):
if (
hasattr(edition, "summary") and len(edition.summary)
)
else (
html.strip_tags(edition.description)
if (
hasattr(edition, "description")
and len(edition.description)
)
else (
html.strip_tags(edition.content[0].value)
if (
hasattr(article, "content")
and len(article.content)
)
else None
)
else html.strip_tags(edition.description)
if (
hasattr(edition, "description")
and len(edition.description)
)
else html.strip_tags(edition.content[0].value)[:200] + "..."
if (
hasattr(article, "content")
and len(article.content)
)
else ""
)
if description:
desc = description[:200] + "..."
else:
desc = ""
description += "..."
instance = models.Edition.objects.create(
title=edition.title,
author_name=author_name,
url=edition.link,
description=desc,
description=description,
updateddate=date_to_tz_aware(edition.updated_parsed),
newsletter=newsletter,
pubdate=date_to_tz_aware(edition.published_parsed),

8
blogs/management/commands/send_weekly_email.py
View file

@ -132,9 +132,13 @@ class Command(BaseCommand):
open_cfps = ""
for instance in cfps:
c_date = instance.closing_date
title_string = f"<h4><a href='{instance.event.url}'>{instance.event.name} - {instance.name}</a></h4>"
title_string = (
f"<h4><a href='{instance.event.url}'>{instance.name}</a></h4>"
)
dates_string = f"<p><strong>Closes:</strong><em>{c_date:%a} {c_date.day} {c_date:%B}</em></p>"
description_string = f"<p>{instance.details}</p><p style='margin-bottom:24px;'>{instance.event.description}</p>"
description_string = (
f"<p style='margin-bottom:24px;'>{instance.details}</p>"
)
string_list = [title_string, dates_string, description_string]
string = "".join(string_list)

3
blogs/models/blog.py
View file

@ -79,7 +79,8 @@ class Blog(BlogData):
author = ""
category = Category(self.category).label
status = f"{self.title}{author} has been added to Aus GLAMR! \n\nIt's about {category}\n\n{self.url}"
status = f"{self.title}{author} has been added to Aus GLAMR! \
\n\nIt's about {category}\n\n{self.url}"
Announcement.objects.create(status=status)
self.announced = True

29
blogs/tests/test_models.py
View file

@ -15,19 +15,10 @@ class BlogTestCase(TestCase):
def setUp(self):
"""set up test blog"""
self.blog = models.Blog.objects.create(
title="my awesome blog",
title="mya awesome blog",
url="https://test.com",
feed="https://test.com/feed.xml",
category="LIB",
author_name="Hugh",
)
self.blog_no_author = models.Blog.objects.create(
title="my awesome archives",
url="https://test2.com",
feed="https://test2.com/feed.xml",
category="ARC",
author_name="",
)
def test_get_absolute_url(self):
@ -55,22 +46,6 @@ class BlogTestCase(TestCase):
self.blog.set_failing()
self.assertEqual(self.blog.failing, True)
def test_announce_blog(self):
"""test announcing the blog"""
self.blog.announce()
status = f"my awesome blog by Hugh has been added to Aus GLAMR! \n\nIt's about Libraries\n\nhttps://test.com"
announcement = models.Announcement.objects.first()
self.assertEqual(status, announcement.status)
def test_announce_blog_no_author(self):
"""test announcing the blog with a blank blog author name"""
self.blog_no_author.announce()
status = f"my awesome archives has been added to Aus GLAMR! \n\nIt's about Archives\n\nhttps://test2.com"
announcement = models.Announcement.objects.first()
self.assertEqual(status, announcement.status)
def test_announce_article(self):
"""announcing a blog article"""
@ -84,7 +59,7 @@ class BlogTestCase(TestCase):
)
article.announce()
status = f"My article (Hugh on my awesome blog)\n\nhttps://example.blog/1"
status = f"My article (Hugh on mya awesome blog)\n\nhttps://example.blog/1"
self.assertTrue(models.Announcement.objects.filter(status=status).exists())

9
blogs/tests/test_utilities.py
View file

@ -70,10 +70,7 @@ class UtilityTests(TestCase):
self.assertEqual(data["description"], "A short summary of my blog")
def test_get_blog_info_no_feed(self):
"""
test get blog info
note get_blog_info only checks <head>, not rss feed
"""
"""test get blog info"""
with open(
pathlib.Path(__file__).parent.joinpath("data/example.html"),
@ -155,7 +152,7 @@ class UtilityTests(TestCase):
data = utilities.get_blog_info("http://test.test")
self.assertEqual(data["title"], "My test website with an RSS feed")
self.assertEqual(data["author_name"], "")
self.assertEqual(data["author_name"], "Hugh Rundle")
self.assertEqual(data["description"], "A short summary of my blog")
def test_get_blog_info_with_incomplete_head_and_partial_feed(self):
@ -174,7 +171,7 @@ class UtilityTests(TestCase):
data = utilities.get_blog_info("http://test.test")
self.assertEqual(data["title"], "My test website with an RSS feed")
self.assertEqual(data["author_name"], "")
self.assertEqual(data["author_name"], None)
self.assertEqual(data["description"], None)
def test_get_webfinger_subscribe_uri(self):

19
blogs/tests/test_views.py
View file

@ -222,7 +222,6 @@ class PublicTests(TestCase):
form.data["from_email"] = "example@example.mail"
form.data["subject"] = "Hello"
form.data["message"] = "Hi there"
form.data["bot_check"] = "books"
request = self.factory.post("contact/", form.data)
request.user = AnonymousUser()
@ -235,24 +234,6 @@ class PublicTests(TestCase):
# Verify that the subject of the first message is correct.
self.assertEqual(mail.outbox[0].subject, "Message via Aus GLAMR: Hello")
def test_contact_from_bot(self):
"""post message"""
view = views.Contact.as_view()
form = forms.ContactForm()
form.data["from_email"] = "example@example.mail"
form.data["subject"] = "Hello"
form.data["message"] = "Hi there"
form.data["bot_check"] = "i am a stupid bot"
request = self.factory.post("contact/", form.data)
request.user = AnonymousUser()
view(request)
# Test that one message has been sent.
self.assertEqual(len(mail.outbox), 0)
def test_search(self):
"""post search query"""

13
blogs/utilities.py
View file

@ -4,7 +4,6 @@ import re
from bs4 import BeautifulSoup
import feedparser
import logging
import requests
from django.conf import settings
@ -66,21 +65,17 @@ def get_blog_info(url):
) # use the title from the feed
blog_info["description"] = description or blog_info.get("description", "")
blog_info["author_name"] = blog_info.get("author", author)
else:
return False # if there is no feed info we need to put the onus back on the user to fill in the data
normalised_author = blog_info["author_name"] or None
if normalised_author:
normalised_author = normalised_author.replace("(noreply@blogger.com)", "")
normalised_author = ""
if author:
normalised_author = author.replace("(noreply@blogger.com)", "")
if normalised_author.strip() == "Unknown":
normalised_author = ""
else:
normalised_author = ""
blog_info["author_name"] = normalised_author
blog_info["author_name"] = normalised_author
return blog_info

8
blogs/views/feeds.py
View file

@ -94,7 +94,9 @@ class EventFeed(Feed):
return (
item.description
if hasattr(item, "description")
else item.details if hasattr(item, "details") else None
else item.details
if hasattr(item, "details")
else None
)
def item_link(self, item):
@ -213,7 +215,9 @@ class CombinedFeed(Feed):
return (
item.description
if hasattr(item, "description")
else item.details if hasattr(item, "details") else None
else item.details
if hasattr(item, "details")
else None
)
def item_link(self, item):

6
blogs/views/public.py
View file

@ -194,9 +194,9 @@ class RegisterBlog(View):
data["blog_info"] = blog_info
else:
data["error"] = (
"Could not auto-discover your feed info, please enter manually"
)
data[
"error"
] = "Could not auto-discover your feed info, please enter manually"
return render(request, "blogs/confirm-register.html", data)

4
docker-compose.yml
View file

@ -1,3 +1,5 @@
version: '3'
services:
db:
image: postgres:13
@ -23,4 +25,4 @@ services:
volumes:
pgdata:
networks:
main:
main:

7
glamr-dev
View file

@ -44,9 +44,6 @@ case "$CMD" in
collectstatic)
runweb python manage.py collectstatic
;;
copystatic)
cp -r static /srv/ausglamr/
;;
createsuperuser)
runweb python manage.py createsuperuser --no-input
;;
@ -77,10 +74,10 @@ case "$CMD" in
runweb python manage.py send_weekly_email
;;
test)
runweb python manage.py test blogs/tests "$@"
runweb python manage.py test "$@"
;;
*)
set +x
echo "That is not a command"
;;
esac
esac

24
nginx.conf.example
View file

@ -1,24 +0,0 @@
server {
server_name example.com;
location = /favicon.ico {
access_log off;
log_not_found off;
}
location /static {
autoindex on;
alias /srv/ausglamr/static;
}
location / {
root html;
index index.html index.htm;
proxy_pass http://127.0.0.1:8080;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}

2
requirements.txt
View file

@ -1,6 +1,6 @@
beautifulsoup4==4.12.2
gunicorn==22.0.0
Django==4.2.14
Django==4.2.11
environs==9.5.0
feedparser==6.0.10
psycopg2==2.9.5