Developing a report of upcoming tasks from WeKan using Jupyter Lab
Specific Solutions uses the self hosted WeKan kanban board. The first step is to update from v2.97 to v3.69 to gat access to improved API docs. Portainer makes this fairly easy, just enter a new docker tag.
Prototyping code
Credentials are kept in gitignored json files as siblings of this notebook
import json
with open(".wekan-creds.json", "r") as creds_file:
creds = json.load(creds_file)
creds.keys()
dict_keys(['username', 'password', 'host'])
# Install a pip package in the current Jupyter kernel
import sys
!{sys.executable} -m pip install requests
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (2.9.1)
import requests
headers = {
'Content-Type': 'application/x-www-form-urlencoded',
'Accept': '*/*'
}
api_key_resp = requests.post(f"{creds['host']}/users/login", {
'username': creds['username'],
'password': creds['password']
}, headers = headers)
api_key = api_key_resp.json()
api_key.keys()
dict_keys(['id', 'token', 'tokenExpires'])
Code readability is important, let's define a method to make it easy to GET from the api
def wekan(url):
return requests.get(f"{creds['host']}{url}", headers = {
'Accept': 'application/json',
'Authorization': f"Bearer {api_key['token']}"
}).json()
user = wekan('/api/user')
user['profile']['fullname']
'Vincent Khougaz'
cards = []
labels = {}
boards = {}
lists = {}
for board_id in user['profile']['starredBoards']:
board = wekan(f"/api/boards/{board_id}")
for label in board["labels"]:
labels[label['_id']] = label['color']
boards[board['_id']] = board
board_lists = wekan(f"/api/boards/{board_id}/lists")
for l in board_lists:
list_id = l["_id"]
lists[list_id] = l
list_cards = wekan(f"/api/boards/{board_id}/lists/{list_id}/cards")
for card in list_cards:
cards.append(wekan(f"/api/boards/{board_id}/lists/{list_id}/cards/{card['_id']}"))
print(len(cards))
print(set(labels.values()))
16
{'blue', 'red', 'purple', 'orange', 'green', 'yellow'}
test_card = wekan(f"/api/boards/8QBuF6qoBT6fsZPeF/lists/3usjdgnA97ZKDboRZ/cards/MSSb4YG9ioYDeHeoP")
test_card
{'_id': 'MSSb4YG9ioYDeHeoP',
'title': 'Productivity Method',
'members': [],
'labelIds': ['Zh2Doq'],
'customFields': [{'_id': 'Ng4gJyj664njtE8Xn', 'value': 3}],
'listId': '3usjdgnA97ZKDboRZ',
'boardId': '8QBuF6qoBT6fsZPeF',
'sort': 0,
'swimlaneId': '7D2Yos4LQAcPLdCRj',
'type': 'cardType-card',
'archived': False,
'parentId': '',
'coverId': '',
'createdAt': '2020-01-14T22:30:30.441Z',
'dateLastActivity': '2020-01-15T03:38:52.473Z',
'description': '',
'requestedBy': '',
'assignedBy': '',
'spentTime': 1,
'isOvertime': False,
'userId': 'uowK3FeqKTf5yx9Ea',
'subtaskSort': -1,
'linkedId': '',
'startAt': None,
'dueAt': '2020-01-15T01:00:00.000Z',
'modifiedAt': '2020-01-15T03:38:52.473Z',
'assignees': []}
from datetime import datetime, timezone
from functools import reduce
has_due = lambda card: 'dueAt' in card and card['dueAt'] is not None
# labels with this color multiply by this much
color_factor = {
'red': 2,
'yellow': 1.5,
'green': 0.75
}
seconds_in_day = 60 * 60 * 24
due_falloff = 10 # not a factor until this long until
due_weight = 5 # when do has this much weight added
def determine_weight(card):
weights = []
if has_due(card):
now = datetime.now(timezone.utc)
due_datetime = datetime.fromisoformat(card['dueAt'].replace("Z", "+00:00"))
days_until = (due_datetime - now).total_seconds() / seconds_in_day
if days_until < 0:
weights.append({
'cause': 'overdue',
'constant': due_weight
})
elif days_until < due_falloff:
weights.append({
'cause': 'due soon',
'constant': max((1 - (days_until / due_falloff)) * due_weight, due_weight)
})
if 'labelIds' in card:
for label_id in card['labelIds']:
if label_id in labels:
color = labels[label_id]
if color in color_factor:
weights.append({
'cause': f"{color} label",
'factor': color_factor[color]
})
break # only the first color matters
return weights
def sum_weight(weight):
constant = sum(w['constant'] for w in weight if 'constant' in w)
factored = reduce(lambda a, b: a * b, (w['factor'] for w in weight if 'factor' in w), constant)
return factored
def describe_weight(weight):
constants = [w for w in weight if 'constant' in w]
constants.sort(key=lambda w: w['constant'])
constants = [f"{w['cause']}({w['constant']:.2f})" for w in constants]
factors = [w for w in weight if 'factor' in w]
factors.sort(key=lambda w: w['factor'])
factors = [f"{w['cause']}({w['factor']}x)" for w in factors]
return f"{' + '.join(constants)} {' '.join(factors)}"
test_weight = determine_weight(test_card)
print(test_weight)
print(sum_weight(test_weight))
print(describe_weight(test_weight))
[{'cause': 'overdue', 'constant': 5}, {'cause': 'red label', 'factor': 2}]
10
overdue(5.00) red label(2x)
priority = sorted(cards, key=lambda c: sum_weight(determine_weight(c)), reverse=True)
for c in priority:
print(f"{c['title']}: {sum_weight(determine_weight(c)):.2f}")
Productivity Method: 10.00
nginx-proxy: 0.00
Certbot Auto Renew: 0.00
Backup Verification: 0.00
heimdall: 0.00
Calendar: 0.00
Software Dashboard: 0.00
Secure Software: 0.00
Hourly Backup of Application Data: 0.00
Document Software: 0.00
Setup Gitlab: 0.00
Bank Account: 0.00
Data Model Diagram: 0.00
XD For 5 Screen: 0.00
Review videos for Round 1: 0.00
Add remaining videos from youtube: 0.00
Here we have the basis of the prioritization system. It assigns a linear factor based on how long until the task is due, and a multiplicitive factor based off how important the task is.
In this way an important task a long way out might have a higher priority than a less important task due soon.
Of course this algorithm will require tweaking, but I'm not going to try to predict the final outcome and instead make a reasonably simple system that is flexible enough to tweak.
Next we need to figure out how to get this sent via email. I have a smtp system available via AWS and can plug straight into that.
with open(".smtp-creds.json", "r") as creds_file:
smtp_creds = json.load(creds_file)
smtp_creds.keys()
dict_keys(['sender', 'receiver', 'username', 'password', 'port', 'host'])
import smtplib
sender = smtp_creds['sender']
receiver = smtp_creds['receiver']
message = f"""From: {sender}
To: Vincent Khougaz <{receiver}>
Subject: Test Email
Email Body!"""
smtp = smtplib.SMTP_SSL(smtp_creds['host'], smtp_creds['port'])
print(smtp.login(smtp_creds['username'], smtp_creds['password']))
smtp.sendmail(sender, [receiver], message)
(235, b'Authentication successful.')
{}
Next we need to figure out how to format the list above into an email. Python string formatting should make this straightforward, if not ergonomic
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
msg = MIMEMultipart()
msg['Subject'] = f"Daily Task Digest for {datetime.now().strftime('%A %Y-%m-%d')}"
msg['From'] = f"Task Digest <{sender}>"
msg['To'] = f"Vincent Khougaz <{receiver}>"
def format_card(card):
weight = determine_weight(card)
return f"""\
<li>
<a
href="{creds['host']}/b/{card['boardId']}/{boards[card['boardId']]['slug']}/{card['_id']}">
{card['title']} ({sum_weight(weight):.2f})
</a>
<br>
<pre>{describe_weight(weight)}</pre>
</li>
"""
html = f"""\
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="https://www.w3.org/1999/xhtml">
<head>
<title>Daily Task Digest for {datetime.now().strftime('%A %Y-%m-%d')}</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0 " />
</head>
<body>
Prioritized tasks for today:
<ul>
{"".join(format_card(c) for c in priority)}
</ul>
</body>
</html>
"""
msg.attach(MIMEText(html, 'html'))
print(msg.as_string())
Content-Type: multipart/mixed; boundary="===============1027601018532405263=="
MIME-Version: 1.0
Subject: Daily Task Digest for Wednesday 2020-01-15
From: Task Digest <task-digest@solutions.land>
To: Vincent Khougaz <vincent@khougaz.com>
--===============1027601018532405263==
Content-Type: text/html; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="https://www.w3.org/1999/xhtml">
<head>
<title>Daily Task Digest for Wednesday 2020-01-15</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0 " />
</head>
<body>
Prioritized tasks for today:
<ul>
<li>
<a
href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/MSSb4YG9ioYDeHeoP">
Productivity Method (10.00)
</a>
<br>
<pre>overdue(5.00) red label(2x)</pre>
</li>
<li>
<a
href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/wgWJqaFLXGGHbfneW">
nginx-proxy (0.00)
</a>
<br>
<pre> </pre>
</li>
<li>
<a
href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/S33QRotdT7yKfCmr9">
Certbot Auto Renew (0.00)
</a>
<br>
<pre> </pre>
</li>
<li>
<a
href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/zgGq2DpeQMF67zSGy">
Backup Verification (0.00)
</a>
<br>
<pre> </pre>
</li>
<li>
<a
href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/D7D8ji9Jd893vuoeY">
heimdall (0.00)
</a>
<br>
<pre> </pre>
</li>
<li>
<a
href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/3LTtc9jfxSspSi942">
Calendar (0.00)
</a>
<br>
<pre> </pre>
</li>
<li>
<a
href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/m29CfHpgefwyBL4z6">
Software Dashboard (0.00)
</a>
<br>
<pre> </pre>
</li>
<li>
<a
href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/kvnzSs7zdDgGcfdid">
Secure Software (0.00)
</a>
<br>
<pre> </pre>
</li>
<li>
<a
href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/DbFdPLqiAntEtP7d5">
Hourly Backup of Application Data (0.00)
</a>
<br>
<pre> </pre>
</li>
<li>
<a
href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/6fD45hKfFhgs9zNtF">
Document Software (0.00)
</a>
<br>
<pre> </pre>
</li>
<li>
<a
href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/mApiEh4yZdHaRJvpD">
Setup Gitlab (0.00)
</a>
<br>
<pre> </pre>
</li>
<li>
<a
href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/K9JGiEeNcC5SdypBd">
Bank Account (0.00)
</a>
<br>
<pre> </pre>
</li>
<li>
<a
href="https://task.solutions.land/b/68H9qWZZ2apetx9nS/offr/WsS63nN3ALwgwprxx">
Data Model Diagram (0.00)
</a>
<br>
<pre> </pre>
</li>
<li>
<a
href="https://task.solutions.land/b/68H9qWZZ2apetx9nS/offr/kjFJt57nAkkR3bdAZ">
XD For 5 Screen (0.00)
</a>
<br>
<pre> </pre>
</li>
<li>
<a
href="https://task.solutions.land/b/R6yMi4SLjGF7A3mAN/content-pipeline/YXGAmRomiixxfwLTK">
Review videos for Round 1 (0.00)
</a>
<br>
<pre> </pre>
</li>
<li>
<a
href="https://task.solutions.land/b/R6yMi4SLjGF7A3mAN/content-pipeline/wh88Yon8a6nt5uf4g">
Add remaining videos from youtube (0.00)
</a>
<br>
<pre> green label(0.75x)</pre>
</li>
</ul>
</body>
</html>
--===============1027601018532405263==--
smtp = smtplib.SMTP_SSL(smtp_creds['host'], smtp_creds['port'])
smtp.login(smtp_creds['username'], smtp_creds['password'])
smtp.sendmail(sender, [receiver], msg.as_string())
{}
Creating a script
Once all of this code has been prototyped it can be collected into a single python file
Note for the future: this file is probably going to be the source of truth for the real deal deployment script. It may end up looking different than the jupyter output above, any deltas you see are the result of updates.
In addition to stripping prints time tracking was added to a final log line, it is useful to have a basic understanding of what's going on with background processes.
$ chmod +x send-digest.py
$ ./send-digest.py
Digest sent for Wednesday 2020-01-15 in 4.127854s
Best practices dictate that this file should be written as a module without import side effects to allow it to be reused as a part of a larger whole.
Instead I didn't. The script works and can be adjusted to account for future needs when those needs are needed.
Deployment
Do the simplest thing. This script will be run by cron and output to a log file. The directory contains secrets, secure it to the user
$ ssh user@deployment-server
$ cd /opt
$ sudo mkdir task-digest
$ sudo chown $(whoami) task-digest
$ sudo chmod 700 task-digest
Experience dictates I will be deploying this script many times.
# deploy.sh
#!/bin/bash
rsync -avh \
send-digest.py \
.smtp-creds.json \
.wekan-creds.json \
user@deployment-server:/opt/task-digest --delete
Making a log file:
sudo touch /var/log/task-digest.log
sudo chown $(whoami) /var/log/task-digest.log
Scheduling the task to run at 8am every day. I can never remember crontab syntax so I use crontab.guru
$ /opt/task-digest/send-digest.py >> /var/log/task-digest.log 2>&1
$ tail /var/log/task-digest.log
FileNotFoundError: [Errno 2] No such file or directory: '.wekan-creds.json'
$ cd /opt/task-digest/ && ./send-digest.py >> /var/log/task-digest.log 2>&1
$ tail -n1 /var/log/task-digest.log
Digest sent for Thursday 2020-01-16 in 1.618252s
$ crontab -e
0 8 * * * cd /opt/task-digest/ && ./send-digest.py >> /var/log/task-digest.log 2>&1
Future improvements
The task ranking algorithm requires tuning, and the email formatting is awkward. It would also be handy to have a webpage that can display this list for both debugging and reference throughout the day.
Either way this is a handy resource to keep myself on track.