New `next_items_page` object for cursor-based pagination

[rachelatmonday (Rachel)]

In the 2023-10 API release, we introduced the items_page object which utilizes cursor-based pagination to retrieve smaller groups of items from a large data set. This object must be nested inside of a boards query, so it has a high complexity cost.

To help reduce that cost, we created the next_items_page object which can be used at the root of your query to paginate through the data set. It takes both the cursor and limit arguments and retrieves the following page of items in the data set.

Now let’s break it down a bit…

Say you only want to retrieve items with a blank status column from board 1234567890. This board contains thousands of items that cannot be retrieved simultaneously, so you can use cursor-based pagination.

The query below would return the ID and name of the first 50 items with a blank status column and a cursor value that represents the position in the data set after returning 50 items.

query {  
  boards (ids:1234567890) {  
    items_page (limit: 50, query_params: {rules: {column_id: "status", compare_value: [5], operator:any_of}}) {  
      cursor  
      items {  
        id  
        name  
      }  
    }  
  }  
}

You can then use the next_items_page object with the cursor argument from the first query to return the following 50 relevant items in the data set. After returning the next cursor value, you can continue paginating through the entire data set.

query {  
  next_items_page (limit: 50, cursor: "MSw5NzI4MDA5MDAsaV9YcmxJb0p1VEdYc1VWeGlxeF9kLDg4MiwzNXw0MTQ1NzU1MTE5") {  
    cursor  
    items {  
      id  
      name  
    }  
  }  
}

[basdebruin]

Hi @rachelatmonday

Love this concept and implemented is straight away. The app did not show expected behavior (my fault) but I think this is worth a warning to fellow developers.

Obviously the JSON data structure returned from the initial call differs from consecutive calls (using next_items_page). If you want to return all data you need to take care of the different data structures. After the initial call I do:

      if (response?.data?.boards[0]?.items_page) {
        cursor = response?.data?.boards[0]?.items_page.cursor;
        resultArray = response?.data?.boards[0]?.items_page.items;
      }

But now (using next_items_page) the follow on calls (assuming you got the cursor) need to be handled a little differently, e.g.:

        cursor = response?.data?.next_items_page.cursor;
        resultArray.push(...response?.data?.next_items_page.items);

Hope this helps others to make use of this great feature.

[Matias.Monday (Matias)]

Thank you @basdebruin for sharing your experience and script!

[Jmiller1 (Jack)]

Hi @basdebruin, I am understanding the structure, the issue I’m having is the response received of the cursor is a string wrapped in single quotes. See picture

This is throwing an error, if I extract the cursor manually and place it in my call wrapped in double quotes it works as intended. Do you have any tips for formatting that correctly within a pagination loop?

I’m working in python and originally saved a variable named cursor to the parse path shown in the picture. Then when I call it in subsequent loops I tried inserting via an f’string which failed. I also tried string concatenation using + operators with the variable called via the str() method. This also produces a string wrapped in single quotes and throws an error. I can’t save double quotes to a variable to insert because I don’t believe python allows you to set those types of operators to a variable. An ideas for what might work?

[basdebruin]

hi @Jmiller1

What I do is very straightforward

      var resultArray = [];
      var cursor = null;

      var query = `query getItemIdsInBoard ($boardId: ID!, $rules: [ItemsQueryRule!])  {
        complexity {
          reset_in_x_seconds
          before
          after
        }
        boards (ids: [$boardId]) {
          items_page (limit:100, query_params: {rules: $rules}) {
            cursor
            items {
              id
              state
              subitems {
                id
              }
            }
          }
        }
      }`;

      var variables = { boardId, rules };
      var response = await mondayClient.api(query, { variables });
      if (response?.data?.boards[0]?.items_page) {
        cursor = response?.data?.boards[0]?.items_page.cursor;
        resultArray = response?.data?.boards[0]?.items_page.items;
      }

      while (cursor) {
        var query = `query getItemIdsInBoardNext ($cursor: String!) {
          complexity {
            reset_in_x_seconds
            before
            after
          }
          next_items_page (limit:100, cursor: $cursor) {
            cursor
            items {
              id
              state
            }
          }
        }`;
        var variables = { cursor };
        var response = await mondayClient.api(query, { variables });
        cursor = response?.data?.next_items_page.cursor;
        resultArray.push(...response?.data?.next_items_page.items);
      }
      return resultArray.length == 0 ? { data: false } : { data: resultArray };

[Jmiller1 (Jack)]

hmmm this is javascript? My structure is very similar and agree using the cursor to access a next page is straight forward. I believe the way python is unpacking the response into json is what’s causing me issues.

Here is what my loop looks like for context:

[basdebruin]

Sure, my example is Javascript. I’m not a Python expert but why do you format the cursor like:

'f"{cursor}"'

can’t you just pass the cursor string you got?

[Jmiller1 (Jack)]

I was not able to and I couldn’t explain exactly why just because I don’t know, but passing the f string like that ignores the " " on the outside of {} and just passes the cursor variable through but the response of the cursor is received like this:

Luckily, I was able to find a way to reformate the quotes like this picture below which when plugged into my loop works as intended. (spent way too much time finding this )

side note, the syntax is a requirement on the graphql side as python classifies these both as strings so most stuff I found were talking about how you can rewrite a string with ’ or " but functionally it doesn’t matter which wasn’t helpful when the string received was already formatted one way and it had to be changed within the code.

Thanks for the responses @basdebruin, talking through always helps me figure stuff out/order my thoughts.

[angel_ch]

Hi @Jmiller1 ,

I am working on a task to query some data with python and have the same issue with passing the cursor string to graphql query request. The cursor string is formatted with double quotes but when plugged in the loop the response still returns error: “Argument ‘cursor’ on Field ‘next_items_page’ has an invalid value (formated_cursor). Expected type ‘String!’.”. Would you mind to share your way of cursor string reference in the query request?

Best,
Angel

[Jmiller1 (Jack)]

Hi @angel_ch,

I had to save it as a reformatted variable. I’ve attached my entire loop below here. Hopefully that will help resolve your issue, but lmk if you have any other questions.

[angel_ch]

Thanks a lot @Jmiller1! It worked like a charm! Btw I would have never guessed the syntax of the cursor reference …

Best,
Angel

[Jmiller1 (Jack)]

Glad it worked! Yeah it took me the better part of a few days to get figured out… was very frustrating

[BruceB (Bruce Baldwin)]

Thanks @Jmiller1 . This helped me out greatly too.