Timeout and data return issues when requesting large queries

I’m running into an interesting issue while performing heavy queries.
If the call takes longer than 60 seconds, I get one of two responses from the server. Both are errors, and both are in HTML body. One is a general 504 timeout warning. The second one is returned the page text below. After watching the timing, this happens at exactly the 60 second mark, so the timeout seems to be 60 seconds from the server. I’m not seeing this mentioned anywhere on the documentation.

–returned page

This exposes a few critical issues and opportunities.

  1. If the anticipated API response body is JSON, ALL API response bodies should be JSON. Every. Single. One. I think it is inappropriate to return a page body when there’s an error, instead of simply {“error”:“Server timeout. Maximum query duration is 60 seconds.”} or something similar displaying the reason for the error. We can always manage exceptions, however response body format should never vary like this.

  2. I’m disappointed further that pagination doesn’t interrupt query timeouts. Return an error at data transmission if anticipated query call will result in greater than 60 seconds, or just complete the query and send the data.

I have noticed similar issues receiving a 504 Timeout Error when trying to retrieve large queries. You bring up very good points about issues and possible opportunities.

I would like to share something that I discovered about query requests that I have not been able to find in the documentation either:
When passing the limit parameter to a query response, subsequent queries to the same board (e.g. within a looping structure) will select the next contiguous block of items from the board, instead of starting from the beginning of the board. I haven’t been able to confirm this, but it seems that Monday.com maintains an internal cursor when a query returns a limited number of items.

When working with large boards (either a large number of items, or a large number of columns), I have found that the ability to “break up” the data that comes from the query solves the problem of the request timeout, and then I can concatenate all results locally.

1 Like

@arnold_wikey yes! I’ve found the same. The two seem to work together so long as they are at the root level of your request. I’m currently pulling full pulse sets across all boards the way you are describing above. I had some weird responses having page requests anywhere except at root during testing that was a bit annoying. For example, I couldn’t successfully pull board --> item with page on item, and of course page on board just breaks up the boards. I think the pagination documentation could use some more description regarding when it does and does not work, as well as documentation regarding time outs and Monday’s recommendation for avoidance.

Hey guys,

there are API rate limits on the complexity levels which are 10,000,000 / minute per account on monday.com.
It’s higly recommended to lower the complexity level of your querys.

You can read more about rate limits here:

@TMNXT-Dev I appreciate your response, however I think we are discussing different topics.

  1. I record the cost of every query I run.
  2. From what I see query costs do not align with machine workload
  3. This is the most expensive query I run. It costs 2,013,010 and runs in less than a second. It costs so much because I pulled a lazy move and didn’t feel like cycling through board page, and Monday’s cost calculator multiplies by 1,000. Regardless, here it is.

  1. The paginated item pulls I am running require much less cost, and still take between 15-45 seconds to run.

  1. Any limit over 2000 on these item requests pushes response time too closely to the 60 second mark for comfort. When I adjust my limit to 4000 or 5000 I bump into response times that are 60+ seconds and run into timeout issues, with a minimal query cost per run time. Here’s an example, using a page number higher than what currently exists in my dataset. Minimal cost, but if the data existed the call is likely to timeout.

{complexity{query, after}, items (page: 50, limit: 5000) {board {id}id,name,state,group {id},column_values {id,text,value}}}
returns: “query”: 190010

  1. When a timeout is achieved, the response is an html page body, instead of a json body with an error code. This add unnecessary complication to error handling.

Hey @morhriveion ,

I don’t represent monday, I’m a dev just like you who wants to help :slight_smile: .

And I’m sorry, I obviously missunderstood your question.

@TMNXT-Dev apologies, I’ll edit my previous response accordingly.

1 Like

Hi @morhriveion!

Scott here from monday.com :wave:

Thank you so much for taking the time to bring this to our attention. The errors being returned in HTML as opposed to JSON is something that we’re aware of and are working on resolving. I’ve passed this feedback along to the team to let them know it’s a pain point that you’re currently running into.

You’d also noted:

I’m disappointed further that pagination doesn’t interrupt query timeouts. Return an error at data transmission if anticipated query call will result in greater than 60 seconds, or just complete the query and send the data.

I just wanted to let you know that I’ve passed that feedback along to the team as well :slight_smile:

If you have any other suggestions, please let me know! :raised_hands: