API performance of heavily used board degrading over time?

Hi,

On my Monday account we have a Monday board which is used to store programatically calculated values. It basically contains for each month the billability percentage of each team in our company. This percentage is recalculated several times a day. When recalculating we delete the original group (e.g. month) and create a new group with new items attached underneath it.

I’m observing that over time the performance of adding items to the groups is getting worse. It now can take more than 40 seconds to add 15 items to a group (each add is 1 mutation API call). When the board was still ‘new’ this only took a few seconds.

Is there an underling reason why performance could be degrading over time in this use case?

Hey @JorneD - could you share the query you’re running to create the items on the group? We’ll take a look on our end and see if we can find anything.

-Daniel

Hi @dsilva, thx for reaching out. Below the details you requested.

The queries run to populate the group of a single month are the following (test-data):

mutation {
    create_item(
      item_name: "billable - test", 
      board_id: 372829960,
      group_id: november75580, 
      column_values: "{\"date4\": {\"date\": \"2020-09-01\"}, \"text0\": \"T&D\", \"numbers\": \"85\" }") {
      id
    }
  }

mutation {
    create_item(
      item_name: "non-billable", 
      board_id: 372829960,
      group_id: november75580, 
      column_values: "{\"date4\": {\"date\": \"2020-09-01\"}, \"text0\": \"T&D\", \"numbers\": \"0\" }") {
      id
    }
  }

mutation {
    create_item(
      item_name: "bench", 
      board_id: 372829960,
      group_id: november75580, 
      column_values: "{\"date4\": {\"date\": \"2020-09-01\"}, \"text0\": \"T&D\", \"numbers\": \"15\" }") {
      id
    }
  }
    mutation {
    create_item(
      item_name: "billable", 
      board_id: 372829960,
      group_id: november75580, 
      column_values: "{\"date4\": {\"date\": \"2020-09-01\"}, \"text0\": \"FINADMIN\", \"numbers\": \"8\" }") {
      id
    }
  }

In total it runs 13 insertions into the same group like this in a loop.

I added some logging to see how long each individual insert takes. It looks like each insert takes around 1.6 seconds. That looks like a lot no? Some log entries below:

2020-11-12T18:05:26.741Z 7bc44eb0-72e3-575a-8d32-1697099cb8b6 INFO mutation { create_item( item_name: "bench", board_id: 372829960, group_id: december22252, column_values: "{\"date4\": {\"date\": \"2020-12-01\"}, \"text0\": \"HR&RECRUITING\", \"numbers\": \"0\" }") { id } }

2020-11-12T18:05:25.315Z 7bc44eb0-72e3-575a-8d32-1697099cb8b6 INFO mutation { create_item( item_name: "non-billable", board_id: 372829960, group_id: december22252, column_values: "{\"date4\": {\"date\": \"2020-12-01\"}, \"text0\": \"HR&RECRUITING\", \"numbers\": \"27\" }") { id } }

2020-11-12T18:05:24.233Z 7bc44eb0-72e3-575a-8d32-1697099cb8b6 INFO mutation { create_item( item_name: "billable", board_id: 372829960, group_id: december22252, column_values: "{\"date4\": {\"date\": \"2020-12-01\"}, \"text0\": \"HR&RECRUITING\", \"numbers\": \"73\" }") { id } }

2020-11-12T18:05:23.157Z 7bc44eb0-72e3-575a-8d32-1697099cb8b6 INFO mutation { create_item( item_name: "bench", board_id: 372

Adding to that, I also noticed that deleting groups from the same board is also very slow, it takes 6 seconds to delete a single group. I will try to run the same code on a new board to validate if it is correlated to the board age/usage.

2020-11-12T19:48:59.953+01:00 2020-11-12T18:48:59.953Z 09a90133-021c-4173-9331-6191a99bc5dc INFO deleting group - november11400

2020-11-12T19:49:06.444+01:00 2020-11-12T18:49:06.444Z 09a90133-021c-4173-9331-6191a99bc5dc INFO deleting group - december6721

2020-11-12T19:49:13.438+01:00 2020-11-12T18:49:13.438Z 09a90133-021c-4173-9331-6191a99bc5dc INFO deleting group - january13896

@dsilva - I ran the same code against a newly created board. The performance difference looks significant:

  • Inserting a new item: 400ms
  • Deleting a group: 1,2s

So I am now quite certain that it is related to the ‘usage history’ of the board. Any thoughts on this?