Mattermost Lunch Channel History - Part I: Data Collection¶

Tasks¶

In this first part of the Mattermost Lunch Channel History Dojo, we aim to retrieve the data from the "lunch" channel for further analysis through the Mattermost API. The concrete goal of this session is to create a file (or multiple files) that contain all of the history of the lunch channel in a suitable format for further analysis (e.g. json or csv). Importantly, the data needs to contain the information, who posted when and what, together with the information who reacted to the post with which emoji(s). :pizza: :+1: :-1: :rocket: :cry:

Setup¶

Installation¶

In [3]:
!pip install --user -q mattermostdriver

WARNING: Ignoring invalid distribution -andas (/usr/local/lib/python3.10/dist-packages)
WARNING: Ignoring invalid distribution -andas (/usr/local/lib/python3.10/dist-packages)

[notice] A new release of pip is available: 24.0 -> 24.1.1
[notice] To update, run: python3 -m pip install --upgrade pip

Package Imports¶

In [4]:
import random
from mattermostdriver import Driver
from getpass import getpass

Group Rotation :memo:¶

In [5]:
# enter the names of your group (comma separated) and update the seed if you like
names = "Sabine,Joél,Marko,Sinta,Jannis"
random.seed(119)
names = names.split(",")
random.shuffle(names)
rotation = " → ".join(names)
rotation
Out[5]:
'Sabine → Sinta → Jannis → Joél → Marko'

Connection to Mattermost¶

In order to access Mattermost, the valid token of a user is required. In order to generate a token, one member of each group needs to download the login.sh script and run it locally. Provide your username and password there, a file .token will be created in your current directory. This token can be entered in the input field that appears when you run the next code cell.

:warning: all API actions are performed as the user who provides the token. Please use it responsibly. The user who created the token should run logout.sh locally, in order to invalidate the token after the session.

In [6]:
mm = Driver({
    'url': 'cctb-intern.biologie.uni-wuerzburg.de',
    'token': getpass("Enter token: "),
    'port': 443})
mm.login()

Enter token:
Out[6]:
{'id': 'wigptpyjapyitjsaffh3x49yzo',
 'create_at': 1587060759230,
 'update_at': 1719999538177,
 'delete_at': 0,
 'username': 'marko_korb',
 'auth_data': '',
 'auth_service': '',
 'email': 'marko.korb@stud-mail.uni-wuerzburg.de',
 'email_verified': True,
 'nickname': '',
 'first_name': 'Marko',
 'last_name': 'Korb',
 'position': '',
 'roles': 'system_user',
 'allow_marketing': True,
 'notify_props': {'auto_responder_active': 'false',
  'auto_responder_message': 'Hello, I am out of office and unable to respond to messages.',
  'channel': 'true',
  'comments': 'never',
  'desktop': 'mention',
  'desktop_notification_sound': 'Bing',
  'desktop_sound': 'true',
  'desktop_threads': 'all',
  'email': 'false',
  'email_threads': 'all',
  'first_name': 'false',
  'mention_keys': 'marko_korb',
  'push': 'mention',
  'push_status': 'online',
  'push_threads': 'all'},
 'last_password_update': 1684227179137,
 'last_picture_update': 1686665165218,
 'locale': 'en',
 'timezone': {'automaticTimezone': 'Europe/Berlin',
  'manualTimezone': '',
  'useAutomaticTimezone': 'true'},
 'disable_welcome_email': False}

Now look at the:

  • Mattermost API documentation
  • Mattermost python driver documentation
In [25]:
def get_posts_for_channel(self, channel_id, params=None):
		return self.client.get(
			Channels.endpoint + '/' + channel_id + '/posts',
			params=params
		)
In [27]:
mm.endpoints.get_posts_for_channel("h3azbb4yejd69kg68kdmqiw53r")

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_738/1693437659.py in <cell line: 1>()
----> 1 mm.endpoints.get_posts_for_channel("h3azbb4yejd69kg68kdmqiw53r")

AttributeError: 'Driver' object has no attribute 'endpoints'
In [24]:
mm.channels.get_channel_by_name("4izweae3eigapetc3b4stqkgna", "lunch")
Out[24]:
{'id': 'h3azbb4yejd69kg68kdmqiw53r',
 'create_at': 1499152209696,
 'update_at': 1689242463847,
 'delete_at': 0,
 'team_id': '4izweae3eigapetc3b4stqkgna',
 'type': 'O',
 'display_name': 'lunch',
 'name': 'lunch',
 'header': '[cooking](https://hackmd.io/PhOpF6VoQN2D9Mg6R3K9bQ?view) - [gather](https://gather.town/app/r4oLVM25VSqeXimY/CCTB)',
 'purpose': 'How to organize lunch :)',
 'last_post_at': 1719910676224,
 'total_msg_count': 6906,
 'extra_update_at': 1533632808244,
 'creator_id': '',
 'scheme_id': None,
 'props': None,
 'group_constrained': None,
 'shared': None,
 'total_msg_count_root': 6971,
 'policy_id': None,
 'last_root_post_at': 1719910676224}
In [22]:
#h3azbb4yejd69kg68kdmqiw53r channeldID
#4izweae3eigapetc3b4stqkgna teamid???
In [21]:
mm.teams.get_team_by_name("CCTB")
Out[21]:
{'id': '4izweae3eigapetc3b4stqkgna',
 'create_at': 1498811058402,
 'update_at': 1558555317773,
 'delete_at': 0,
 'display_name': 'CCTB',
 'name': 'cctb',
 'description': '',
 'email': '',
 'type': 'O',
 'company_name': '',
 'allowed_domains': '',
 'invite_id': 'yg6m1911ot8djkajhqgigfw6xy',
 'allow_open_invite': False,
 'last_team_icon_update': 1558555317773,
 'scheme_id': None,
 'group_constrained': None,
 'policy_id': None,
 'cloud_limits_archived': False}
In [31]:
posts = mm.posts.get_posts_for_channel("h3azbb4yejd69kg68kdmqiw53r")
In [36]:
len(posts["posts"])
Out[36]:
60
In [37]:
posts.keys()
Out[37]:
dict_keys(['order', 'posts', 'next_post_id', 'prev_post_id', 'has_next', 'first_inaccessible_post_time'])
In [38]:
posts['posts'].keys()
Out[38]:
dict_keys(['13yo1tsrnid15bd3tohgqgnash', '1f77hkb68py5ukuzfdcw1j8mty', '35tzkq6b7j8wj8tyq14etq9bka', '3jqtuj71qpda8dse6tjnhdeq4c', '5hbfj51hjfrsprqq8koacfk18w', '6emh6hsoqpngpg4jrqktyexiyc', '76e1tuu3efyyzxr4x85mibimno', '79jpo46j1igrmgkdfy5zrozshc', '7btah8qzjtdipchgz5fi38ykbc', '7ks7xffjbf81mc6c9zf5hknhxe', '81budfh9siygdyamjubf8ei3oo', '87kzhpk9w3yuf8676sex4iytjw', '8t5od64petrs7yfyktfxfjrwdh', '9364e7fsapyqbk1hk86io1bk4o', '9rm7m18jcibm3rbef3yqm5biyy', '9w8iujqjjfy97gc7augi7dd3pa', 'b86aige1e3y1byju3mr76do33e', 'b894tc3hzid1uknu54c1tub7ta', 'bc95qih3hbnejgrtfgkx4tqzda', 'dbbfdznta7fadbcrf9rqn16y7r', 'dgmy447pxpg8zjwfhthokcmx6c', 'dguym6hfs7ba883ns6g36j819c', 'dwyheudckbn7ddpagcc7qzjewo', 'dz4hzybdd7n63rmaocwpoqiear', 'e7qy74cyypd5urb6mtc8173t1h', 'ecipwk81y3biird76adshrybqo', 'f1eex4cugtyrzr86uoq5ca9geh', 'f59jj8myn7bg8n1j1kejp3u6ch', 'fd41dxodoi8t5bza39koap6tbc', 'fj56yogwxidk3p8b4yehb96cte', 'fwmouzqgdf87bk9633tn8tez5o', 'gyrs5ykyutywpyie43hhnwnjfa', 'gzd3gqkr7indiffu64k3rsfe5y', 'iyj1hy18ytnb9d1z6enx3fpaaa', 'izhcm4y6bibwpknf5kxk86g6ir', 'k7og97nx1jn9pxk4jhxmytnneo', 'kkwpm969zpd5jpnpwp11d86mrh', 'm684im46ypbjdcqoi5ims3ipkr', 'mxzgiwe4ef8tpfznhn1bwuswyy', 'nuxrfkcd5ffn9czdf7w37szhdc', 'o6aymh945irkjxre9dtwjck4ec', 'odruwa49wjr4ucrjoca6i9dihr', 'onypbwh1opbdbdfbn9n7uwbmna', 'oo7zsxa1yffbx86hsdfwuyutmw', 'oqwejddezjrbjcd9k5gsww81jo', 'pzitxo18afb6pgxcka1r9ywn8h', 's9d136m3abyhibeerdu46twswo', 't6xcmaj6fp8z7fftfiauyr4t3y', 'tbiapa4kfiydbxg333r3coyukc', 'wghc6zfpztg9xjp71ketfgbypo', 'wmgxbhn6p3de7mb7btkhp845xw', 'wngtjqih4ibodkb4yq1wpiitkw', 'x3mrxg1jctr1tki489ncaoo3ah', 'xm4h713rybnp8kfcgsk8wbz5dr', 'y9oprc85rjfedfomzcrteu1bye', 'ysd8mo34ptfijg6p8p3mp4heqh', 'ysrq8a5f1tfmxnmace4m6ptbec', 'ywkwbkbbp3rx8prdk6gzqnm9ih', 'yy66qb9ifpn33gb5frt566ukse', 'zcaptninj3gzmc11fk93sek9sa'])
In [43]:
posts['posts']['oqwejddezjrbjcd9k5gsww81jo']
Out[43]:
{'id': 'oqwejddezjrbjcd9k5gsww81jo',
 'create_at': 1719565810471,
 'update_at': 1719842238888,
 'edit_at': 0,
 'delete_at': 0,
 'is_pinned': False,
 'user_id': 'xxg7ytrh9idsdci9zo3mjqq96e',
 'channel_id': 'h3azbb4yejd69kg68kdmqiw53r',
 'root_id': '',
 'original_id': '',
 'message': '11:15?',
 'type': '',
 'props': {'disable_group_highlight': True},
 'hashtags': '',
 'file_ids': ['3jtgydibdtddukoes6qpe5q8eh'],
 'pending_post_id': '',
 'has_reactions': True,
 'reply_count': 0,
 'last_reply_at': 0,
 'participants': None,
 'metadata': {'emojis': [{'id': 'fupxasprbjr6x8x8j9d6wnbtga',
    'create_at': 1575379253544,
    'update_at': 1575379253544,
    'delete_at': 0,
    'creator_id': 'zactjsrscf8ippuib6ui63sfbc',
    'name': 'cherry'}],
  'files': [{'id': '3jtgydibdtddukoes6qpe5q8eh',
    'user_id': 'xxg7ytrh9idsdci9zo3mjqq96e',
    'post_id': 'oqwejddezjrbjcd9k5gsww81jo',
    'channel_id': 'h3azbb4yejd69kg68kdmqiw53r',
    'create_at': 1719565804717,
    'update_at': 1719565804717,
    'delete_at': 0,
    'name': 'Bildschirmfoto 2024-06-28 um 11.09.35.png',
    'extension': 'png',
    'size': 1061331,
    'mime_type': 'image/png',
    'width': 736,
    'height': 736,
    'has_preview_image': True,
    'mini_preview': '/9j/2wCEAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsKCwsNDhIQDQ4RDgsLEBYQERMUFRUVDA8XGBYUGBIUFRQBAwQEBQQFCQUFCRQNCw0UFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFP/AABEIABAAEAMBIgACEQEDEQH/xAGiAAABBQEBAQEBAQAAAAAAAAAAAQIDBAUGBwgJCgsQAAIBAwMCBAMFBQQEAAABfQECAwAEEQUSITFBBhNRYQcicRQygZGhCCNCscEVUtHwJDNicoIJChYXGBkaJSYnKCkqNDU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6g4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2drh4uPk5ebn6Onq8fLz9PX29/j5+gEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoLEQACAQIEBAMEBwUEBAABAncAAQIDEQQFITEGEkFRB2FxEyIygQgUQpGhscEJIzNS8BVictEKFiQ04SXxFxgZGiYnKCkqNTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqCg4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2dri4+Tl5ufo6ery8/T19vf4+fr/2gAMAwEAAhEDEQA/APl/RtIsJvFOt6OYftV1dnMM0rR+fJGxcyx5IkRXYEYI+YFflIBIH0J8Af2cdA0nxp4R1u402Yabqck99CdcgBkAhTfglQFKtJ8m3JOB82Nyivnf4fJ4Wkjudd1nxg9ndHy1utGmUKNQDHcSJHBjQIQvDc56YAzXtN/8Z/A3iDSbaKwuLrwxqVlA9vZ38OtxyGJd4dyqSXBUs+zHC4IA2gcGvjK3PBKELu277v8Ar5fM/RVJV5zmrRT0tpovLz/4Y//Z',
    'remote_id': '',
    'archived': False}],
  'reactions': [{'user_id': 'e964eohjctd7xe7e96adyxw5do',
    'post_id': 'oqwejddezjrbjcd9k5gsww81jo',
    'emoji_name': 'cherry',
    'create_at': 1719565848046,
    'update_at': 1719565848046,
    'delete_at': 0,
    'remote_id': '',
    'channel_id': 'h3azbb4yejd69kg68kdmqiw53r'},
   {'user_id': 'ajs477544if7ip5r5yawkitf3e',
    'post_id': 'oqwejddezjrbjcd9k5gsww81jo',
    'emoji_name': 'cherry',
    'create_at': 1719842238882,
    'update_at': 1719842238882,
    'delete_at': 0,
    'remote_id': '',
    'channel_id': 'h3azbb4yejd69kg68kdmqiw53r'}]}}
In [0]: