MirrorValue drastically increasing response times

I’ve got a query that I am running, in which there are a large amount of Mirror columns. Previously, I was able to fetch all of this data using the text field. However, introducing unravelling MirrorValue increases the time of this request from 2 seconds to 7.5 seconds while still only querying a single column. It looks like this continues to compound with more data and columns.
The data stored in the MirrorValue is required to resolve the value of the column.

Am I doing this completely wrong? This seems like a massive regression with the new API. In addition, since the pagination requires a cursor from the previous response, this drastically increases the net time to paginate through a board as concurrent queries are not possible.

In addition, it doesn’t look like I can actually grab the text values of Dropdowns at all with this new system.

Old Query (2 seconds):

"{boards(ids:[BOARD_ID]){items (limit: 50, page: 1) {name, group(){title} column_values(ids: [mirror1, dropdown, shift_days1, shift0, sample_column, mirror42, mirror82, mirror3, check4, data_catalog_id]) {id, title, text}}}}"

New Query, Single Column, No MirrorValue (2 seconds):

"{boards(ids:[BOARD_ID]){items_page (limit: 50) {cursor items { name column_values ( ids: [\"mirror1\"]){column {id} id type value }}}}}"

New Query, Single Column, Expand MirrorValue (11 seconds):

"{boards(ids:[BOARD_ID]){items_page (limit: 50) {cursor items { name column_values ( ids: [\"mirror1\"]){column {id} id type value ...on MirrorValue{ text: display_value} }}}}}"

New Query, All Columns, No MirrorValue (2 seconds):

"{boards(ids:BOARD_ID){items_page (limit: 50) {cursor items { name group {title} column_values ( ids: [\"mirror1\",\"dropdown\",\"shift_days1\",\"shift0\",\"sample_column\",\"mirror42\",\"mirror82\",\"mirror3\",\"check4\",\"data_catalog_id\"]){column {id} id type value }}}}}"

New Query, All Columns, Expand MirrorValue (40 seconds):

"{boards(ids:[BOARD_ID]){items_page (limit: 50) {cursor items { name group {title} column_values ( ids: [\"mirror1\",\"dropdown\",\"shift_days1\",\"shift0\",\"sample_column\",\"mirror42\",\"mirror82\",\"mirror3\",\"check4\",\"data_catalog_id\"]){column {id} id type value ...on MirrorValue{ display_value} }}}}}"

This is a known issue with the display_value of mirror columns and they are actively working on a fix.

Just an FYI previously with dropdowns the value also only provided the index. You could get text values within the text field of the column value and this still works. You could never get them in a structured value in the old API (you could match them up with an index via the settings_str of the column though). With the new API for dropdowns you can do something like this and actually return the selected dropdown values with the id and label.

{
  items(ids: "123342342"){
    column_values(ids: "dropdown"){
      id
      type
      text
      value
      ... on DropdownValue {
        values {
          id
          label
        }
      }
    }
  }
}

One method of speeding up pagination in the new API is to return just items and their IDs (and maybe names), avoiding column values which require additional lookups by the GraphQL resolvers. You can get up to 500 at a time this way relatively quickly.

Then you can parallelize requesting the full data with simple items(ids: [ids]) queries, which you can do batches of 100 and execute 5 at a time. Simultaneously, you can be fetching the next batch of IDs.

Not the ideal solution, but it should speed things up. That said, for your main performance issue, wait and test periodically over the next week or two and you may find they’ve solved it hopefully.

1 Like

Awesome, thanks for the fast response. I missed the DropdownValue, thanks for the details there.

I’ll wait for the MirrorValue performance to get better, good to know this is on the team’s radar.

1 Like

It seems like this was fixed temporarily yesterday, but it has regressed back. Do we have an idea of
A. When the testing environment will be stable
B. When this will be fixed?

We are getting pretty close to the holiday season and I’m getting worried about the implementation details here.

1 Like

So I’m understanding the optimization that you described, it would be the following:

  1. Fetch all items on a board, but only grab their IDs
  2. Create x parallel requests out to resolve batches of y IDs at a time

The issue I see with this is that it doesn’t seem like this is something you actively endorse (Items)

The data I’m trying to grab here is so simple I feel like I must be missing an endpoint. Is there a way to quickly grab the visible data on a given board quickly?

For reference, it takes around 3 seconds per 100 items returned without MirrorValue. At 2000 items in the board (which is a relatively small amount of data), this is a minute of response time. This was acceptable before because we could parallelize the query, but the recent changes have forced us to be serial.

I don’t work for monday :slight_smile: so I am not the one saying its “slower”.

The documentation there is misleading I think. The reason using boards.items_page.items is “faster” than items directly is root objects are limited to 100 IDs, and items_page.items is limited to 500 items per requests. Therefore fewer requests = faster. Before items_page, boards.items was 1000 ids limit. So yes, faster in the sense of fewer requests (assuming you didn’t parallelize them).

The other issue is the old database (pre-mondaydb) indexed data differently than the current database, with items being indexed with the board. I don’t believe the new database has that structural limitation (being schema-less)

The slow down with MirrorValue is an issue I know they are actively working on and found the root cause to, and issued a fix they had to roll back because of a regression. Once this is fixed you can use the items_page to iterate the whole board, in a few seconds for each request again.

1 Like

Ah, didn’t realize you were just helping out, apologies. Let me see if I can get my company’s rep involved.

Thanks for the help on this!

1 Like

Hello there,

Our team is working on a fix for this that should improve the response time for queries like this one.

I do not have an ETA yet, but we will probably announce when we deploy it.

Cheers,
Matias

1 Like

Looks like the fix is live, and I was able to get response times down to somewhat reasonable limits (~12 seconds for 500 items).

Thanks for the help everyone!

1 Like

Happy to hear that!

1 Like