AboutServicesProjectsBlog
Projects
Protoyping with python by creating an email task digest

Protoyping with python by creating an email task digest

2020-1-13

Developing a report of upcoming tasks from WeKan using Jupyter Lab

Specific Solutions uses the self hosted WeKan kanban board. The first step is to update from v2.97 to v3.69 to gat access to improved API docs. Portainer makes this fairly easy, just enter a new docker tag.

Prototyping code

Credentials are kept in gitignored json files as siblings of this notebook

[80]
import json

with open(".wekan-creds.json", "r") as creds_file:
    creds = json.load(creds_file)

creds.keys()
dict_keys(['username', 'password', 'host'])
[4]
# Install a pip package in the current Jupyter kernel
import sys
!{sys.executable} -m pip install requests
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (2.9.1)

Using the excellent requests library and the wekan api

[81]
import requests

headers = {
  'Content-Type': 'application/x-www-form-urlencoded',
  'Accept': '*/*'
}

api_key_resp = requests.post(f"{creds['host']}/users/login", {
    'username': creds['username'],
    'password': creds['password']
}, headers = headers)

api_key = api_key_resp.json()
api_key.keys()
dict_keys(['id', 'token', 'tokenExpires'])

Code readability is important, let's define a method to make it easy to GET from the api

[6]
def wekan(url):
    return requests.get(f"{creds['host']}{url}", headers = {
        'Accept': 'application/json',
        'Authorization': f"Bearer {api_key['token']}"
    }).json()

user = wekan('/api/user')
user['profile']['fullname']
'Vincent Khougaz'
[92]
cards = []
labels = {}
boards = {}
lists = {}

for board_id in user['profile']['starredBoards']:
    board = wekan(f"/api/boards/{board_id}")
    for label in board["labels"]:
        labels[label['_id']] = label['color']
    boards[board['_id']] = board
    board_lists = wekan(f"/api/boards/{board_id}/lists")
    for l in board_lists:
        list_id = l["_id"]
        lists[list_id] = l
        list_cards = wekan(f"/api/boards/{board_id}/lists/{list_id}/cards")
        for card in list_cards:
            cards.append(wekan(f"/api/boards/{board_id}/lists/{list_id}/cards/{card['_id']}"))

print(len(cards))
print(set(labels.values()))
16 {'blue', 'red', 'purple', 'orange', 'green', 'yellow'}
[11]
test_card = wekan(f"/api/boards/8QBuF6qoBT6fsZPeF/lists/3usjdgnA97ZKDboRZ/cards/MSSb4YG9ioYDeHeoP")
test_card
{'_id': 'MSSb4YG9ioYDeHeoP',
 'title': 'Productivity Method',
 'members': [],
 'labelIds': ['Zh2Doq'],
 'customFields': [{'_id': 'Ng4gJyj664njtE8Xn', 'value': 3}],
 'listId': '3usjdgnA97ZKDboRZ',
 'boardId': '8QBuF6qoBT6fsZPeF',
 'sort': 0,
 'swimlaneId': '7D2Yos4LQAcPLdCRj',
 'type': 'cardType-card',
 'archived': False,
 'parentId': '',
 'coverId': '',
 'createdAt': '2020-01-14T22:30:30.441Z',
 'dateLastActivity': '2020-01-15T03:38:52.473Z',
 'description': '',
 'requestedBy': '',
 'assignedBy': '',
 'spentTime': 1,
 'isOvertime': False,
 'userId': 'uowK3FeqKTf5yx9Ea',
 'subtaskSort': -1,
 'linkedId': '',
 'startAt': None,
 'dueAt': '2020-01-15T01:00:00.000Z',
 'modifiedAt': '2020-01-15T03:38:52.473Z',
 'assignees': []}
[89]
from datetime import datetime, timezone
from functools import reduce

has_due = lambda card: 'dueAt' in card and card['dueAt'] is not None

# labels with this color multiply by this much
color_factor = {
    'red': 2,
    'yellow': 1.5,
    'green': 0.75
}

seconds_in_day = 60 * 60 * 24
due_falloff = 10 # not a factor until this long until
due_weight = 5 # when do has this much weight added

def determine_weight(card):
    weights = []
    
    if has_due(card):
        now = datetime.now(timezone.utc)
        due_datetime = datetime.fromisoformat(card['dueAt'].replace("Z", "+00:00"))
        days_until = (due_datetime - now).total_seconds() / seconds_in_day
        if days_until < 0:
            weights.append({
                'cause': 'overdue',
                'constant': due_weight
            })
        elif days_until < due_falloff:
            weights.append({
                'cause': 'due soon',
                'constant': max((1 - (days_until / due_falloff)) * due_weight, due_weight)
            })

    if 'labelIds' in card:
        for label_id in card['labelIds']:
            if label_id in labels:
                color = labels[label_id]
                if color in color_factor:
                    weights.append({
                        'cause': f"{color} label",
                        'factor': color_factor[color]
                    })
                    break # only the first color matters
    
    return weights
    
def sum_weight(weight):
    constant = sum(w['constant'] for w in weight if 'constant' in w)
    factored = reduce(lambda a, b: a * b, (w['factor'] for w in weight if 'factor' in w), constant)
    
    return factored

def describe_weight(weight):
    constants = [w for w in weight if 'constant' in w]
    constants.sort(key=lambda w: w['constant'])
    constants = [f"{w['cause']}({w['constant']:.2f})" for w in constants]
    factors = [w for w in weight if 'factor' in w]
    factors.sort(key=lambda w: w['factor'])
    factors = [f"{w['cause']}({w['factor']}x)" for w in factors]

    return f"{' + '.join(constants)} {' '.join(factors)}"

test_weight = determine_weight(test_card)
print(test_weight)
print(sum_weight(test_weight))
print(describe_weight(test_weight))
[{'cause': 'overdue', 'constant': 5}, {'cause': 'red label', 'factor': 2}] 10 overdue(5.00) red label(2x)
[93]
priority = sorted(cards, key=lambda c: sum_weight(determine_weight(c)), reverse=True)

for c in priority:
    print(f"{c['title']}: {sum_weight(determine_weight(c)):.2f}")
Productivity Method: 10.00 nginx-proxy: 0.00 Certbot Auto Renew: 0.00 Backup Verification: 0.00 heimdall: 0.00 Calendar: 0.00 Software Dashboard: 0.00 Secure Software: 0.00 Hourly Backup of Application Data: 0.00 Document Software: 0.00 Setup Gitlab: 0.00 Bank Account: 0.00 Data Model Diagram: 0.00 XD For 5 Screen: 0.00 Review videos for Round 1: 0.00 Add remaining videos from youtube: 0.00

Here we have the basis of the prioritization system. It assigns a linear factor based on how long until the task is due, and a multiplicitive factor based off how important the task is.

In this way an important task a long way out might have a higher priority than a less important task due soon.

Of course this algorithm will require tweaking, but I'm not going to try to predict the final outcome and instead make a reasonably simple system that is flexible enough to tweak.

Next we need to figure out how to get this sent via email. I have a smtp system available via AWS and can plug straight into that.

[27]
with open(".smtp-creds.json", "r") as creds_file:
    smtp_creds = json.load(creds_file)

smtp_creds.keys()
dict_keys(['sender', 'receiver', 'username', 'password', 'port', 'host'])
[30]
import smtplib

sender = smtp_creds['sender']
receiver = smtp_creds['receiver']

message = f"""From: {sender}
To: Vincent Khougaz <{receiver}>
Subject: Test Email

Email Body!"""

smtp = smtplib.SMTP_SSL(smtp_creds['host'], smtp_creds['port'])
print(smtp.login(smtp_creds['username'], smtp_creds['password']))

smtp.sendmail(sender, [receiver], message)
(235, b'Authentication successful.')
{}

Next we need to figure out how to format the list above into an email. Python string formatting should make this straightforward, if not ergonomic

[95]
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

msg = MIMEMultipart()
msg['Subject'] = f"Daily Task Digest for {datetime.now().strftime('%A %Y-%m-%d')}"
msg['From'] = f"Task Digest <{sender}>"
msg['To'] = f"Vincent Khougaz <{receiver}>"

def format_card(card):
    weight = determine_weight(card)
    return f"""\
        <li>
            <a
                href="{creds['host']}/b/{card['boardId']}/{boards[card['boardId']]['slug']}/{card['_id']}">
                {card['title']} ({sum_weight(weight):.2f})
            </a>
            <br>
            <pre>{describe_weight(weight)}</pre>
        </li>
"""

html = f"""\
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="https://www.w3.org/1999/xhtml">
<head>
<title>Daily Task Digest for {datetime.now().strftime('%A %Y-%m-%d')}</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0 " />
</head>
<body>
Prioritized tasks for today:
<ul>
{"".join(format_card(c) for c in priority)}
</ul>
</body>
</html>
"""

msg.attach(MIMEText(html, 'html'))

print(msg.as_string())
Content-Type: multipart/mixed; boundary="===============1027601018532405263==" MIME-Version: 1.0 Subject: Daily Task Digest for Wednesday 2020-01-15 From: Task Digest <task-digest@solutions.land> To: Vincent Khougaz <vincent@khougaz.com> --===============1027601018532405263== Content-Type: text/html; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="https://www.w3.org/1999/xhtml"> <head> <title>Daily Task Digest for Wednesday 2020-01-15</title> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <meta http-equiv="X-UA-Compatible" content="IE=edge" /> <meta name="viewport" content="width=device-width, initial-scale=1.0 " /> </head> <body> Prioritized tasks for today: <ul> <li> <a href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/MSSb4YG9ioYDeHeoP"> Productivity Method (10.00) </a> <br> <pre>overdue(5.00) red label(2x)</pre> </li> <li> <a href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/wgWJqaFLXGGHbfneW"> nginx-proxy (0.00) </a> <br> <pre> </pre> </li> <li> <a href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/S33QRotdT7yKfCmr9"> Certbot Auto Renew (0.00) </a> <br> <pre> </pre> </li> <li> <a href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/zgGq2DpeQMF67zSGy"> Backup Verification (0.00) </a> <br> <pre> </pre> </li> <li> <a href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/D7D8ji9Jd893vuoeY"> heimdall (0.00) </a> <br> <pre> </pre> </li> <li> <a href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/3LTtc9jfxSspSi942"> Calendar (0.00) </a> <br> <pre> </pre> </li> <li> <a href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/m29CfHpgefwyBL4z6"> Software Dashboard (0.00) </a> <br> <pre> </pre> </li> <li> <a href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/kvnzSs7zdDgGcfdid"> Secure Software (0.00) </a> <br> <pre> </pre> </li> <li> <a href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/DbFdPLqiAntEtP7d5"> Hourly Backup of Application Data (0.00) </a> <br> <pre> </pre> </li> <li> <a href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/6fD45hKfFhgs9zNtF"> Document Software (0.00) </a> <br> <pre> </pre> </li> <li> <a href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/mApiEh4yZdHaRJvpD"> Setup Gitlab (0.00) </a> <br> <pre> </pre> </li> <li> <a href="https://task.solutions.land/b/8QBuF6qoBT6fsZPeF/specific-solutions/K9JGiEeNcC5SdypBd"> Bank Account (0.00) </a> <br> <pre> </pre> </li> <li> <a href="https://task.solutions.land/b/68H9qWZZ2apetx9nS/offr/WsS63nN3ALwgwprxx"> Data Model Diagram (0.00) </a> <br> <pre> </pre> </li> <li> <a href="https://task.solutions.land/b/68H9qWZZ2apetx9nS/offr/kjFJt57nAkkR3bdAZ"> XD For 5 Screen (0.00) </a> <br> <pre> </pre> </li> <li> <a href="https://task.solutions.land/b/R6yMi4SLjGF7A3mAN/content-pipeline/YXGAmRomiixxfwLTK"> Review videos for Round 1 (0.00) </a> <br> <pre> </pre> </li> <li> <a href="https://task.solutions.land/b/R6yMi4SLjGF7A3mAN/content-pipeline/wh88Yon8a6nt5uf4g"> Add remaining videos from youtube (0.00) </a> <br> <pre> green label(0.75x)</pre> </li> </ul> </body> </html> --===============1027601018532405263==--
[90]
smtp = smtplib.SMTP_SSL(smtp_creds['host'], smtp_creds['port'])
smtp.login(smtp_creds['username'], smtp_creds['password'])
smtp.sendmail(sender, [receiver], msg.as_string())
{}

Creating a script

Once all of this code has been prototyped it can be collected into a single python file

Note for the future: this file is probably going to be the source of truth for the real deal deployment script. It may end up looking different than the jupyter output above, any deltas you see are the result of updates.

In addition to stripping prints time tracking was added to a final log line, it is useful to have a basic understanding of what's going on with background processes.

$ chmod +x send-digest.py
$ ./send-digest.py
Digest sent for Wednesday 2020-01-15 in 4.127854s

Best practices dictate that this file should be written as a module without import side effects to allow it to be reused as a part of a larger whole.

Instead I didn't. The script works and can be adjusted to account for future needs when those needs are needed.

Deployment

Do the simplest thing. This script will be run by cron and output to a log file. The directory contains secrets, secure it to the user

$ ssh user@deployment-server
$ cd /opt
$ sudo mkdir task-digest
$ sudo chown $(whoami) task-digest
$ sudo chmod 700 task-digest

Experience dictates I will be deploying this script many times.

# deploy.sh

#!/bin/bash

rsync -avh \
    send-digest.py \
    .smtp-creds.json \
    .wekan-creds.json \
    user@deployment-server:/opt/task-digest --delete

Making a log file:

sudo touch /var/log/task-digest.log
sudo chown $(whoami) /var/log/task-digest.log

Scheduling the task to run at 8am every day. I can never remember crontab syntax so I use crontab.guru

$ /opt/task-digest/send-digest.py >> /var/log/task-digest.log 2>&1
$ tail /var/log/task-digest.log
FileNotFoundError: [Errno 2] No such file or directory: '.wekan-creds.json'

$ cd /opt/task-digest/ && ./send-digest.py >> /var/log/task-digest.log 2>&1
$ tail -n1 /var/log/task-digest.log
Digest sent for Thursday 2020-01-16 in 1.618252s

$ crontab -e
0 8 * * * cd /opt/task-digest/ && ./send-digest.py >> /var/log/task-digest.log 2>&1

Future improvements

The task ranking algorithm requires tuning, and the email formatting is awkward. It would also be handy to have a webpage that can display this list for both debugging and reference throughout the day.

Either way this is a handy resource to keep myself on track.

Related Projects:
Mr Fixums' Lathe Handwheel RepairFirst Welding Project: Making a Weld CartBuilding a Generator in Several Distinct StepsMachining a Tube BenderUpgrading Shop Air CompressorPicture FramingQuick fix: Fixing household goods with a 3d printerMachining a camera mountHanging PlotterBuilding the Official Prusa Printer Enclosure out of Ikea Lack Tables
Featured Projects
GeneratorHandwheel Repair
Company Info
About UsContactAffiliate DisclosurePrivacy Policy
Specific Solutions LLC
Portland, OR